Optimal Scheduling of Entropy Regularizer for Continuous-Time Linear-Quadratic Reinforcement Learning

DOI10.1137/22M1515744arXiv2208.04466WikidataQ129754889 ScholiaQ129754889MaRDI QIDQ6180253FDOQ6180253

Lukasz Szpruch, Tanut Treetanthiploet, Yu-Fei Zhang

Publication date: 19 January 2024

Published in: SIAM Journal on Control and Optimization (Search for Journal in Brave)

Abstract: This work uses the entropy-regularised relaxed stochastic control perspective as a principled framework for designing reinforcement learning (RL) algorithms. Herein agent interacts with the environment by generating noisy controls distributed according to the optimal relaxed policy. The noisy policies, on the one hand, explore the space and hence facilitate learning but, on the other hand, introduce bias by assigning a positive probability to non-optimal actions. This exploration-exploitation trade-off is determined by the strength of entropy regularisation. We study algorithms resulting from two entropy regularisation formulations: the exploratory control approach, where entropy is added to the cost objective, and the proximal policy update approach, where entropy penalises the divergence of policies between two consecutive episodes. We analyse the finite horizon continuous-time linear-quadratic (LQ) RL problem for which both algorithms yield a Gaussian relaxed policy. We quantify the precise difference between the value functions of a Gaussian policy and its noisy evaluation and show that the execution noise must be independent across time. By tuning the frequency of sampling from relaxed policies and the parameter governing the strength of entropy regularisation, we prove that the regret, for both learning algorithms, is of the order $mathcalO(sqrtN)$ (up to a logarithmic factor) over $N$ episodes, matching the best known result from the literature.

Full work available at URL: https://arxiv.org/abs/2208.04466

zbMATH Keywords

linear-quadraticentropy regularizationregret analysiscontinuous-time reinforcement learningexploratory controlproximal policy update

Mathematics Subject Classification ID

Sequential statistical design (62L05) Measures of information, entropy (94A17) Linear-quadratic optimal control problems (49N10) Stochastic learning and adaptive control (93E35)

Cites Work

• Title not available (Why is that?)
• High-Dimensional Probability
• Mirror descent and nonlinear projected subgradient methods for convex optimization.
• Title not available (Why is that?)
• Title not available (Why is that?)
• Compactification methods in the control of degenerate diffusions: existence of an optimal control
• Title not available (Why is that?)
• Title not available (Why is that?)
• The exact law of large numbers via Fubini extension and characterization of insurable risks
• Adaptive continuous-time linear quadratic Gaussian control
• On Incomplete Learning and Certainty-Equivalence Control
• Reinforcement learning and stochastic optimisation
• Title not available (Why is that?)
• On the sample complexity of the linear quadratic regulator
• Continuous‐time mean–variance portfolio selection: A reinforcement learning framework
• Exploratory LQG mean field games with entropy regularization
• A modified MSA for stochastic control problems
• Entropy Regularization for Mean Field Games with Learning
• Regularity and Stability of Feedback Relaxed Controls

Cited In (1)

• Logarithmic regret bounds for continuous-time average-reward Markov decision processes

This page was built for publication: Optimal Scheduling of Entropy Regularizer for Continuous-Time Linear-Quadratic Reinforcement Learning

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6180253)