Smoothing policies and safe policy gradients

DOI10.1007/S10994-022-06232-6MaRDI QIDQ6097096zbMATH OpenOpenAlexFDO

Authors Matteo Papini, Matteo Pirotta, Marcello Restelli

Publication date 12 June 2023

Published in Machine Learning (Search for Journal in Brave)

Copyright license Creative Commons Attribution 4.0 International

Full work available at URL https://arxiv.org/abs/1905.03231

zbMATH Keywords

monotonic improvementpolicy gradientreinforcement learningsafe learning

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05)

Abstract: Policy Gradient (PG) algorithms are among the best candidates for the much-anticipated applications of reinforcement learning to real-world control tasks, such as robotics. However, the trial-and-error nature of these methods poses safety issues whenever the learning process itself must be performed on a physical system or involves any form of human-computer interaction. In this paper, we address a specific safety formulation, where both goals and dangers are encoded in a scalar reward signal and the learning agent is constrained to never worsen its performance, measured as the expected sum of rewards. By studying actor-only policy gradient from a stochastic optimization perspective, we establish improvement guarantees for a wide class of parametric policies, generalizing existing results on Gaussian policies. This, together with novel upper bounds on the variance of policy gradient estimators, allows us to identify meta-parameter schedules that guarantee monotonic improvement with high probability. The two key meta-parameters are the step size of the parameter updates and the batch size of the gradient estimates. Through a joint, adaptive selection of these meta-parameters, we obtain a policy gradient algorithm with monotonic improvement guarantees.

Cites work

• A comprehensive survey on safe reinforcement learning
• A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play
• A Stochastic Approximation Method
• Analysis and improvement of policy gradient estimation
• Approximate policy iteration: a survey and some new methods
• Compatible natural gradient policy search
• Discounted Markov decision processes with utility constraints
• Expected policy gradients for reinforcement learning
• scientific article; zbMATH DE number 1804128 (Why is no real title available?)
• scientific article; zbMATH DE number 7225149 (Why is no real title available?)
• scientific article; zbMATH DE number 1753152 (Why is no real title available?)
• scientific article; zbMATH DE number 796853 (Why is no real title available?)
• scientific article; zbMATH DE number 5685899 (Why is no real title available?)
• scientific article; zbMATH DE number 2243395 (Why is no real title available?)
• OnActor-Critic Algorithms
• Online portfolio selection: a survey
• Policy gradient in Lipschitz Markov decision processes
• Probability theory. A comprehensive course.
• Reinforcement learning. An introduction
• Safe policy iteration: a monotonically improving approximate policy iteration approach
• Simple statistical gradient-following algorithms for connectionist reinforcement learning
• Stochastic optimal control. The discrete time case

Cited in

(2)

• Global convergence of natural policy gradient with Hessian-aided momentum variance reduction
• Search or split: policy gradient with adaptive policy space

This page was built for publication: Smoothing policies and safe policy gradients

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6097096)