Dynamic Programming Principles for Mean-Field Controls with Learning
From MaRDI portal
Abstract: Dynamic programming principle (DPP) is fundamental for control and optimization, including Markov decision problems (MDPs), reinforcement learning (RL), and more recently mean-field controls (MFCs). However, in the learning framework of MFCs, DPP has not been rigorously established, despite its critical importance for algorithm designs. In this paper, we first present a simple example in MFCs with learning where DPP fails with a mis-specified Q function; and then propose the correct form of Q function in an appropriate space for MFCs with learning. This particular form of Q function is different from the classical one and is called the IQ function. In the special case when the transition probability and the reward are independent of the mean-field information, it integrates the classical Q function for single-agent RL over the state-action distribution. In other words, MFCs with learning can be viewed as lifting the classical RLs by replacing the state-action space with its probability distribution space. This identification of the IQ function enables us to establish precisely the DPP in the learning framework of MFCs. Finally, we illustrate through numerical experiments the time consistency of this IQ function.
Recommendations
- Model-free mean-field reinforcement learning: mean-field MDP and mean-field Q-learning
- Dynamic programming for mean-field type control
- Continuous time q-learning for mean-field control problems
- Unified reinforcement Q-learning for mean field game and control problems
- A framework for the dynamic programming principle and martingale-generated control correspondences
Cited in
(7)- Recent developments in machine learning methods for stochastic control and games
- Infinite horizon average cost optimality criteria for mean-field control
- Continuous time q-learning for mean-field control problems
- Fast policy learning for linear-quadratic control with entropy regularization
- Time-inconsistent mean-field stopping problems: a regularised equilibrium approach
- Learning with linear function approximations in mean-field control
- Finite approximations for mean-field type multi-agent control and their near optimality
This page was built for publication: Dynamic Programming Principles for Mean-Field Controls with Learning
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6192781)
