Policy optimization for H₂ linear control with H_ robustness guarantee: implicit regularization and global convergence
From MaRDI portal
Publication:5163686
Abstract: Policy optimization (PO) is a key ingredient for reinforcement learning (RL). For control design, certain constraints are usually enforced on the policies to optimize, accounting for either the stability, robustness, or safety concerns on the system. Hence, PO is by nature a constrained (nonconvex) optimization in most cases, whose global convergence is challenging to analyze in general. More importantly, some constraints that are safety-critical, e.g., the -norm constraint that guarantees the system robustness, are difficult to enforce as the PO methods proceed. Recently, policy gradient methods have been shown to converge to the global optimum of linear quadratic regulator (LQR), a classical optimal control problem, without regularizing/projecting the control iterates onto the stabilizing set, its (implicit) feasible set. This striking result is built upon the coercive property of the cost, ensuring that the iterates remain feasible as the cost decreases. In this paper, we study the convergence theory of PO for linear control with -norm robustness guarantee. One significant new feature of this problem is the lack of coercivity, i.e., the cost may have finite value around the feasible set boundary, breaking the existing analysis for LQR. Interestingly, we show that two PO methods enjoy the implicit regularization property, i.e., the iterates preserve the robustness constraint as if they are regularized by the algorithms. Furthermore, despite the nonconvexity of the problem, we show that these algorithms converge to the globally optimal policies with globally sublinear rates, avoiding all suboptimal stationary points/local minima, and with locally (super-)linear rates under certain conditions.
Recommendations
- New insight into the simultaneous policy update algorithms related to \(H_\infty\) state feedback control
- Robust reinforcement learning for stochastic linear quadratic control with multiplicative noise
- Global convergence of policy gradient methods to (almost) locally optimal policies
- Fast global convergence of natural policy gradient methods with entropy regularization
- Nonconvex policy search using variational inequalities
Cites work
- \(H^ \infty\)-optimal control and related minimax design problems. A dynamic game approach.
- A new CAD method and associated architectures for linear controllers
- An exact solution to general four-block discrete-time mixed ℋ/sub 2//ℋ/sub ∞/ problems via convex optimization
- Computational methods for parametric LQ problems--A survey
- Convergence and Sample Complexity of Gradient Methods for the Model-Free Linear–Quadratic Regulator Problem
- Cubic regularization of Newton method and its global performance
- Derivative-free methods for policy optimization: guarantees for linear quadratic systems
- Existence and comparison theorems for algebraic Riccati equations for continuous- and discrete-time systems
- H/sub ∞/ design with pole placement constraints: an LMI approach
- scientific article; zbMATH DE number 1001726 (Why is no real title available?)
- scientific article; zbMATH DE number 48691 (Why is no real title available?)
- scientific article; zbMATH DE number 1026562 (Why is no real title available?)
- scientific article; zbMATH DE number 1765103 (Why is no real title available?)
- scientific article; zbMATH DE number 1418964 (Why is no real title available?)
- Linear Matrix Inequalities in System and Control Theory
- LQG control with an H/sup infinity / performance bound: a Riccati equation approach
- Matrix differential calculus with applications to simple, Hadamard, and Kronecker products
- Minimum entropy \(H_{\infty}\) control
- Mixed H₂/H_\infty Control via Nonsmooth Optimization
- Mixed \({\mathcal H}_ 2/{\mathcal H}_{\infty}\) control for discrete-time systems via convex optimization
- Mixed H/sub 2//H/sub infinity / control: a convex optimization approach
- Mixed-norm \(H_ 2/H_{\infty}\) regulation and estimation: The discrete- time case
- Multiobjective H/sub 2//H/sub ∞/ control
- On the complexity of steepest descent, Newton's and regularized Newton's methods for nonconvex unconstrained optimization problems
- On the Kalman-Yakubovich-Popov lemma
- OnActor-Critic Algorithms
- Optimal stochastic linear systems with exponential performance criteria and their relation to deterministic differential games
- State-space formulae for all stabilizing controllers that satisfy an \(H_{\infty}\)-norm bound and relations to risk sensitivity
Cited in
(13)- Optimizing static linear feedback: gradient method
- A comparison of guaranteeing and Kalman filters
- Decentralized nonconvex optimization with guaranteed privacy and accuracy
- Minimum input design for direct data-driven property identification of unknown linear systems
- Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergence
- Convergence of policy gradient methods for finite-horizon exploratory linear-quadratic control problems
- Model-free approximate dynamic programming for stochastic zero-sum games: algorithm design and analysis
- Fast policy learning for linear-quadratic control with entropy regularization
- Learning zero-sum linear quadratic games with improved sample complexity and last-iterate convergence
- Policy gradient converges to the globally optimal policy for nearly linear-quadratic regulators
- Policy optimization of finite-horizon Kalman filter with unknown noise covariance
- Ergodic-risk criterion for stochastically stabilizing policy optimization
- On model-free learning over rate-limited channels: the linear quadratic regulator case
This page was built for publication: Policy optimization for \(\mathcal{H}_2\) linear control with \(\mathcal{H}_\infty\) robustness guarantee: implicit regularization and global convergence
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5163686)
