Optimal Scheduling of Entropy Regularizer for Continuous-Time Linear-Quadratic Reinforcement Learning

DOI10.1137/22M1515744MaRDI QIDQ6180253zbMATH OpenWikidataFDO

Authors Lukasz Szpruch, Tanut Treetanthiploet, Yu-Fei Zhang

Publication date 19 January 2024

Published in SIAM Journal on Control and Optimization (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2208.04466

zbMATH Keywords

linear-quadraticentropy regularizationregret analysiscontinuous-time reinforcement learningexploratory controlproximal policy update

Mathematics Subject Classification ID

Sequential statistical design (62L05) Measures of information, entropy (94A17) Linear-quadratic optimal control problems (49N10) Stochastic learning and adaptive control (93E35)

Abstract: This work uses the entropy-regularised relaxed stochastic control perspective as a principled framework for designing reinforcement learning (RL) algorithms. Herein agent interacts with the environment by generating noisy controls distributed according to the optimal relaxed policy. The noisy policies, on the one hand, explore the space and hence facilitate learning but, on the other hand, introduce bias by assigning a positive probability to non-optimal actions. This exploration-exploitation trade-off is determined by the strength of entropy regularisation. We study algorithms resulting from two entropy regularisation formulations: the exploratory control approach, where entropy is added to the cost objective, and the proximal policy update approach, where entropy penalises the divergence of policies between two consecutive episodes. We analyse the finite horizon continuous-time linear-quadratic (LQ) RL problem for which both algorithms yield a Gaussian relaxed policy. We quantify the precise difference between the value functions of a Gaussian policy and its noisy evaluation and show that the execution noise must be independent across time. By tuning the frequency of sampling from relaxed policies and the parameter governing the strength of entropy regularisation, we prove that the regret, for both learning algorithms, is of the order $mathcalO(sqrtN)$ (up to a logarithmic factor) over $N$ episodes, matching the best known result from the literature.

Cites work

• scientific article; zbMATH DE number 3567644 (Why is no real title available?)
• scientific article; zbMATH DE number 1325009 (Why is no real title available?)
• scientific article; zbMATH DE number 7307478 (Why is no real title available?)
• scientific article; zbMATH DE number 3283642 (Why is no real title available?)
• A modified MSA for stochastic control problems
• Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems
• Adaptive continuous-time linear quadratic Gaussian control
• Compactification methods in the control of degenerate diffusions: existence of an optimal control
• Continuous‐time mean–variance portfolio selection: A reinforcement learning framework
• Entropy Regularization for Mean Field Games with Learning
• Exploratory LQG mean field games with entropy regularization
• High-dimensional probability. An introduction with applications in data science
• Mirror descent and nonlinear projected subgradient methods for convex optimization.
• On incomplete learning and certainty-equivalence control
• On the sample complexity of the linear quadratic regulator
• Regularity and stability of feedback relaxed controls
• Reinforcement learning and stochastic optimisation
• Reinforcement learning. An introduction
• The exact law of large numbers via Fubini extension and characterization of insurable risks

Cited in

(10)

• Actor-critic learning algorithms for mean-field control with moment neural networks
• Logarithmic regret bounds for continuous-time average-reward Markov decision processes
• Continuous-time risk-sensitive reinforcement learning via quadratic variation penalty
• Continuous-time optimal investment with portfolio constraints: a reinforcement learning approach
• Continuous time reinforcement learning: a random measure approach
• Fast policy learning for linear-quadratic control with entropy regularization
• Mirror descent for stochastic control problems with measure-valued controls
• Two system transformation data-driven algorithms for linear quadratic mean-field games
• Sublinear regret for a class of continuous-time linear-quadratic reinforcement learning problems
• Entropy annealing for policy mirror descent in continuous time and space

This page was built for publication: Optimal Scheduling of Entropy Regularizer for Continuous-Time Linear-Quadratic Reinforcement Learning

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6180253)