The Dynamics of Q-learning in Population Games: a Physics-Inspired Continuity Equation Model, Shuyue Hu★, Chin-Wing Leung, Ho-fung Leung, and Harold Soh★, International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2022
Links:
Although learning has found wide application in multi-agent systems, its effects on the temporal evolution of a system are far from understood.
This paper focuses on the dynamics of Q-learning in large-scale multi-agent systems modeled as population games. We revisit the replicator equation model for Q-learning dynamics and observe that this model is inappropriate for our concerned setting. Motivated by this, we develop a new formal model, which bears a formal connection with the continuity equation in physics. We show that our model always accurately describes the Q-learning dynamics in population games across different initial settings of MASs and game configurations. We also show that our model can be applied to different exploration mechanisms, describe the mean dynamics, and be extended to Q-learning in 2-player and n-player games. Last but not least, we show that our model can provide insights into algorithm parameters and facilitate parameter tuning.
Resources
You can find our paper here.
Citation
Please consider citing our paper if you build upon our results and ideas.
Shuyue Hu★, Chin-Wing Leung, Ho-fung Leung, and Harold Soh★, “The Dynamics of Q-learning in Population Games: a Physics-Inspired Continuity Equation Model”, International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2022
@inproceedings{hu2022,
author = {Shuyue Hu and Chin-Wing Leung and Ho-fung Leung and Harold Soh},
title = {The Dynamics of Q-learning in Population Games: a Physics-Inspired Continuity Equation Model},
year = {2022},
booktitle = {International Conference on Autonomous Agents and Multiagent Systems (AAMAS)}}
Contact
If you have questions or comments, please contact Shuyue Hu.
Acknowledgements
This research is supported by the National Research Foundation Singapore under its AI Singapore Programme (Award Number: AISG-RP-2019-011).
