Graphon mean-field control for cooperative multi-agent reinforcement learning

DOI10.1016/J.JFRANKLIN.2023.09.002MaRDI QIDQ6136535zbMATH OpenOpenAlexFDO

Authors Yuanquan Hu, Xiaoli Wei, Junji Yan, Hengxi Zhang

Publication date 17 January 2024

Published in Journal of the Franklin Institute (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2209.04808

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05) Applications of graph theory (05C90) Multi-agent systems (93A16)

Abstract: The marriage between mean-field theory and reinforcement learning has shown a great capacity to solve large-scale control problems with homogeneous agents. To break the homogeneity restriction of mean-field theory, a recent interest is to introduce graphon theory to the mean-field paradigm. In this paper, we propose a graphon mean-field control (GMFC) framework to approximate cooperative multi-agent reinforcement learning (MARL) with nonuniform interactions and show that the approximate order is of $mathcalO(frac1sqrtN)$, with $N$ the number of agents. By discretizing the graphon index of GMFC, we further introduce a smaller class of GMFC called block GMFC, which is shown to well approximate cooperative MARL. Our empirical studies on several examples demonstrate that our GMFC approach is comparable with the state-of-art MARL algorithms while enjoying better scalability.

Recommendations

• Mean-field controls with Q-learning for cooperative MARL: convergence and complexity analysis
• Model-free mean-field reinforcement learning: mean-field MDP and mean-field Q-learning
• Graphon mean field games and their equations
• Unified reinforcement Q-learning for mean field game and control problems
• A General Framework for Learning Mean-Field Games

Cites work

• Convergence of weighted empirical measures
• Distributed learning and cooperative control for multi-agent systems
• Finite mean field games: fictitious play and convergence to a first order continuous mean field game
• Graphon Control of Large-Scale Networks of Linear Systems
• High-dimensional statistics. A non-asymptotic viewpoint
• Large networks and graph limits
• Large population stochastic dynamic games: closed-loop McKean-Vlasov systems and the Nash certainty equivalence principle
• Mean field game of controls and an application to trade crowding
• Mean field games
• Mean field games: A toy model on an Erdös-Renyi graph.
• Mean-field controls with Q-learning for cooperative MARL: convergence and complexity analysis
• Probabilistic analysis of mean-field games
• Stochastic graphon games. I: The static case

Cited in

(1)

• Breaking consensus in kinetic opinion formation models on graphons

This page was built for publication: Graphon mean-field control for cooperative multi-agent reinforcement learning

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6136535)