Compatible natural gradient policy search

DOI10.1007/S10994-019-05807-0MaRDI QIDQ2320577zbMATH OpenOpenAlexWikidataFDO

Authors Joni Pajarinen, Hong Linh Thai, Riad Akrour, Jan Peters, Gerhard Neumann

Publication date 23 August 2019

Published in Machine Learning (Search for Journal in Brave)

Copyright license Creative Commons Attribution 4.0 International

Full work available at URL https://arxiv.org/abs/1902.02823

zbMATH Keywords

policy searchreinforcement learning

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05) Problem solving in the context of artificial intelligence (heuristics, search strategies, etc.) (68T20) Markov and semi-Markov decision processes (90C40)

Abstract: Trust-region methods have yielded state-of-the-art results in policy search. A common approach is to use KL-divergence to bound the region of trust resulting in a natural gradient policy update. We show that the natural gradient and trust region optimization are equivalent if we use the natural parameterization of a standard exponential policy distribution in combination with compatible value function approximation. Moreover, we show that standard natural gradient updates may reduce the entropy of the policy according to a wrong schedule leading to premature convergence. To control entropy reduction we introduce a new policy search method called compatible policy search (COPOS) which bounds entropy loss. The experimental results show that COPOS yields state-of-the-art results in challenging continuous control tasks and in discrete partially observable tasks.

Recommendations

• Approximate Newton Policy Gradient Algorithms
• Policy gradient in continuous time
• Fast global convergence of natural policy gradient methods with entropy regularization
• Natural actor-critic algorithms
• Expected policy gradients for reinforcement learning

Cites work

• Hierarchical relative entropy policy search
• scientific article; zbMATH DE number 6982305 (Why is no real title available?)
• scientific article; zbMATH DE number 2107836 (Why is no real title available?)
• Natural evolution strategies
• Online planning algorithms for POMDPS
• Simple statistical gradient-following algorithms for connectionist reinforcement learning
• The cross-entropy method for combinatorial and continuous optimization

Cited in

(4)

• Convex optimization with an interpolation-based projection and its application to deep learning
• 10.1162/jmlr.2003.3.4-5.921
• A stochastic trust-region framework for policy optimization
• Smoothing policies and safe policy gradients

Describes a project that uses

Uses Software

• POMDPS
• CARLA

This page was built for publication: Compatible natural gradient policy search

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2320577)