Explicit explore, exploit, or escape (E^4): near-optimal safety-constrained reinforcement learning in polynomial time
From MaRDI portal
Publication:6106432
Abstract: In reinforcement learning (RL), an agent must explore an initially unknown environment in order to learn a desired behaviour. When RL agents are deployed in real world environments, safety is of primary concern. Constrained Markov decision processes (CMDPs) can provide long-term safety constraints; however, the agent may violate the constraints in an effort to explore its environment. This paper proposes a model-based RL algorithm called Explicit Explore, Exploit, or Escape (), which extends the Explicit Explore or Exploit () algorithm to a robust CMDP setting. explicitly separates exploitation, exploration, and escape CMDPs, allowing targeted policies for policy improvement across known states, discovery of unknown states, as well as safe return to known states. robustly optimises these policies on the worst-case CMDP from a set of CMDP models consistent with the empirical observations of the deployment environment. Theoretical results show that finds a near-optimal constraint-satisfying policy in polynomial time whilst satisfying safety constraints throughout the learning process. We then discuss as a practical algorithmic framework, including robust-constrained offline optimisation algorithms, the design of uncertainty sets for the transition dynamics of unknown states, and how to further leverage empirical observations and prior knowledge to relax some of the worst-case assumptions underlying the theory.
Recommendations
- Safe Exploration of State and Action Spaces in Reinforcement Learning
- Safe policy iteration: a monotonically improving approximate policy iteration approach
- Safety-constrained reinforcement learning with a distributional safety critic
- Safe exploration in model-based reinforcement learning using control barrier functions
- Safe Policies for Reinforcement Learning via Primal-Dual Methods
- Safe reinforcement learning for continuous spaces through Lyapunov-constrained behavior
- Planning for potential: efficient safe reinforcement learning
- Safe Reinforcement Learning Using Robust MPC
- Safe reinforcement learning: A control barrier function optimization approach
Cites work
- 10.1162/153244303765208377
- \({\mathcal Q}\)-learning
- A block coordinate descent method for regularized multiconvex optimization with applications to nonnegative tensor factorization and completion
- A new polynomial-time algorithm for linear programming
- Constrained Markov decision processes with total cost criteria: Lagrangian approach and dual linear program
- scientific article; zbMATH DE number 3644821 (Why is no real title available?)
- scientific article; zbMATH DE number 1348599 (Why is no real title available?)
- scientific article; zbMATH DE number 1453042 (Why is no real title available?)
- Interior-point methods
- Near-optimal regret bounds for reinforcement learning
- Near-optimal reinforcement learning in polynomial time
- Probability Inequalities for Sums of Bounded Random Variables
- Robust Control of Markov Decision Processes with Uncertain Transition Matrices
- Robust Dynamic Programming
- Robust Markov Decision Processes
Cited in
(2)
This page was built for publication: Explicit explore, exploit, or escape \((E^4)\): near-optimal safety-constrained reinforcement learning in polynomial time
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6106432)
