New Versions of Gradient Temporal-Difference Learning

DOI10.1109/TAC.2022.3213763MaRDI QIDQ6093230zbMATH OpenFDO

Authors

Publication date 6 October 2023

Published in IEEE Transactions on Automatic Control (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2109.04033

zbMATH Keywords

convergenceoptimizationreinforcement learning (RL)saddle-point problemstabilitytemporal-difference (TD) learning

Mathematics Subject Classification ID

Systems theory; control (93-XX)

Abstract: Sutton, Szepesv'{a}ri and Maei introduced the first gradient temporal-difference (GTD) learning algorithms compatible with both linear function approximation and off-policy training. The goal of this paper is (a) to propose some variants of GTDs with extensive comparative analysis and (b) to establish new theoretical analysis frameworks for the GTDs. These variants are based on convex-concave saddle-point interpretations of GTDs, which effectively unify all the GTDs into a single framework, and provide simple stability analysis based on recent results on primal-dual gradient dynamics. Finally, numerical comparative analysis is given to evaluate these approaches.

Recommendations

• Differential Temporal Difference Learning
• Technical update: Least-squares temporal difference learning
• Factored temporal difference learning in the New Ties environment
• Hyperbolically Discounted Temporal Difference Learning
• Generalized TD learning
• Practical issues in temporal difference learning
• Linear least-squares algorithms for temporal difference learning

Cited in

(1)

• Hyperbolically Discounted Temporal Difference Learning

This page was built for publication: New Versions of Gradient Temporal-Difference Learning

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6093230)