On-line policy gradient estimation with multi-step sampling

DOI10.1007/S10626-009-0078-3MaRDI QIDQ5962027zbMATH OpenOpenAlexFDO

Authors Yanjie Li, Fang Cao, Xi-Ren Cao

Publication date 16 September 2010

Published in Discrete Event Dynamic Systems (Search for Journal in Brave)

Full work available at URL https://doi.org/10.1007/s10626-009-0078-3

zbMATH Keywords

Markov reward processeson-line estimationperformance potentials

Mathematics Subject Classification ID

Markov processes: estimation; hidden Markov models (62M05)

Review text

The authors discuss the problem of sample-path-based (on-line) performance gradient estimation for Markov systems. The existing on-line performance gradient estimation algorithms generally require a standard importance sampling assumption. Examples are given to illustrate that the existing on-line policy gradient approaches cannot provide an accurate gradient estimate when the assumption does not hold. It is shown that this assumption can be relaxed and a few new algorithms are proposed based on multi-step sampling. These algorithms do not require this assumption. All the algorithms can be implemented on sample paths and policy gradients can be estimated on-line.

Recommendations

• scientific article; zbMATH DE number 1961138
• scientific article; zbMATH DE number 1753153
• A basic formula for performance gradient estimation of semi-Markov decision processes
• Approximate gradient methods in policy-space optimization of Markov reward processes
• Performance gradient estimation for the very large finite Markov chains

Cites work

• A basic formula for online policy gradient algorithms
• scientific article; zbMATH DE number 3532286 (Why is no real title available?)
• scientific article; zbMATH DE number 700091 (Why is no real title available?)
• scientific article; zbMATH DE number 1753152 (Why is no real title available?)
• scientific article; zbMATH DE number 1753153 (Why is no real title available?)
• Perturbation realization, potentials, and sensitivity analysis of Markov processes
• Simulation-based optimization of Markov reward processes
• Variance reduction techniques for gradient estimates in reinforcement learning

Cited in

(3)

• scientific article; zbMATH DE number 1961138 (Why is no real title available?)
• An Incremental Fast Policy Search Using a Single Sample Path
• Online Bootstrap Inference For Policy Evaluation In Reinforcement Learning

This page was built for publication: On-line policy gradient estimation with multi-step sampling

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5962027)