Two Time-Scale Stochastic Approximation with Controlled Markov Noise and Off-Policy Temporal-Difference Learning

DOI10.1287/moor.2017.0855zbMath1434.62174arXiv1503.09105MaRDI QIDQ5219302

Shalabh Bhatnagar, Prasenjit Karmakar

Publication date: 11 March 2020

Published in: Mathematics of Operations Research (Search for Journal in Brave)

Full work available at URL: https://arxiv.org/abs/1503.09105

zbMATH Keywords

asymptotic convergence; temporal-difference learning; Markov noise; two time-scale stochastic approximation

Mathematics Subject Classification ID

60J05: Discrete-time Markov processes on general state spaces

62L20: Stochastic approximation

93E35: Stochastic learning and adaptive control

Related Items

A Two-Timescale Stochastic Algorithm Framework for Bilevel Optimization: Complexity Analysis and Application to Actor-Critic, A Two-Time-Scale Stochastic Optimization Framework with Applications in Control and Reinforcement Learning, Whittle index based Q-learning for restless bandits with average reward, On Generalized Bellman Equations and Temporal-Difference Learning, Stochastic Recursive Inclusions in Two Timescales with Nonadditive Iterate-Dependent Markov Noise

Cites Work

• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Stochastic approximation with two time scales
• Convergence and convergence rate of stochastic gradient search in the case of multiple and non-isolated extrema
• Basis function adaptation in temporal difference reinforcement learning
• Linear stochastic approximation driven by slowly varying Markov chains
• Stochastic approximation with `controlled Markov' noise
• Stochastic approximations for finite-state Markov chains
• Applications of a Kushner and Clark lemma to general classes of stochastic algorithms
• OnActor-Critic Algorithms
• Least Squares Temporal Difference Methods: An Analysis under General Conditions
• Stochastic Approximations and Differential Inclusions