Model-based policy gradients with parameter-based exploration by least-squares conditional density estimation

DOI10.1016/J.NEUNET.2014.06.006MaRDI QIDQ889297zbMATH OpenOpenAlexWikidataFDO

Authors Voot Tangkaratt, Syogo Mori, Tingting Zhao, Jun Morimoto, Masashi Sugiyama

Publication date 6 November 2015

Published in Neural Networks (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1307.5118

zbMATH Keywords

conditional density estimationreinforcement learningtransition model estimation

Mathematics Subject Classification ID

Density estimation (62G07) Learning and adaptive systems in artificial intelligence (68T05)

Abstract: The goal of reinforcement learning (RL) is to let an agent learn an optimal control policy in an unknown environment so that future expected rewards are maximized. The model-free RL approach directly learns the policy based on data samples. Although using many samples tends to improve the accuracy of policy learning, collecting a large number of samples is often expensive in practice. On the other hand, the model-based RL approach first estimates the transition model of the environment and then learns the policy based on the estimated transition model. Thus, if the transition model is accurately learned from a small amount of data, the model-based approach can perform better than the model-free approach. In this paper, we propose a novel model-based RL method by combining a recently proposed model-free policy search method called policy gradients with parameter-based exploration and the state-of-the-art transition model estimator called least-squares conditional density estimation. Through experiments, we demonstrate the practical usefulness of the proposed method.

Recommendations

• Model-based reinforcement learning with dimension reduction
• Efficient sample reuse in policy gradients with parameter-based exploration
• Learning parametric policies and transition probability models of Markov decision processes from data
• Integrating a partial model into model free reinforcement learning
• Model-based reinforcement learning for approximate optimal regulation

Cites work

• 10.1162/1532443041827907
• A least-squares approach to direct importance estimation
• Adaptive importance sampling for value function approximation in off-policy reinforcement learning
• Analysis and improvement of policy gradient estimation
• Computational complexity of kernel-based density-ratio estimation: a condition number analysis
• Density-ratio matching under the Bregman divergence: a unified framework of density-ratio estimation
• Efficient exploration through active learning for value function approximation in reinforcement learning
• Efficient sample reuse in policy gradients with parameter-based exploration
• Gaussian processes for machine learning.
• scientific article; zbMATH DE number 1373584 (Why is no real title available?)
• Model-based contextual policy search for data-efficient generalization of robot skills
• Policy search for motor primitives in robotics
• Simple statistical gradient-following algorithms for connectionist reinforcement learning
• Statistical analysis of kernel-based least-squares density-ratio estimation
• Sufficient dimension reduction via squared-loss mutual information estimation
• Using Expectation-Maximization for Reinforcement Learning

Cited in

(4)

• Model-based reinforcement learning with dimension reduction
• Integrating a partial model into model free reinforcement learning
• Operator shifting for model-based policy evaluation
• Policy search for active fault diagnosis with partially observable state

Describes a project that uses

Uses Software

• PILCO

This page was built for publication: Model-based policy gradients with parameter-based exploration by least-squares conditional density estimation

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q889297)