Approximate Q Learning for Controlled Diffusion Processes and Its Near Optimality

DOI10.1137/22M1484201zbMATH Open1521.93214arXiv2203.07499OpenAlexW4385162439MaRDI QIDQ6136230FDOQ6136230

Authors: Erhan Bayraktar, Ali Devran Kara 

Publication date: 29 August 2023

Published in: SIAM Journal on Mathematics of Data Science (Search for Journal in Brave)

Abstract: We study a Q learning algorithm for continuous time stochastic control problems. The proposed algorithm uses the sampled state process by discretizing the state and control action spaces under piece-wise constant control processes. We show that the algorithm converges to the optimality equation of a finite Markov decision process (MDP). Using this MDP model, we provide an upper bound for the approximation error for the optimal value function of the continuous time control problem. Furthermore, we present provable upper-bounds for the performance loss of the learned control process compared to the optimal admissible control process of the original problem. The provided error upper-bounds are functions of the time and space discretization parameters, and they reveal the effect of different levels of the approximation: (i) approximation of the continuous time control problem by an MDP, (ii) use of piece-wise constant control processes, (iii) space discretization. Finally, we state a time complexity bound for the proposed algorithm as a function of the time and space discretization parameters.

Full work available at URL: https://arxiv.org/abs/2203.07499

Recommendations

• Q-learning for continuous-time linear systems: A model-free infinite horizon optimal control approach
• Convergence of discretization procedure in \(Q\)-learning
• Convergence of a Q-learning variant for continuous states and actions
• Integral \(Q\)-learning and explorized policy iteration for adaptive optimal control of continuous-time linear systems
• A Q-Learning Algorithm for Discrete-Time Linear-Quadratic Control with Random Parameters of Unknown Distribution: Convergence and Stabilization

zbMATH Keywords

diffusion processesreinforcement learningMDP (Markov decision process)

Mathematics Subject Classification ID

Diffusion processes (60J60) Markov and semi-Markov decision processes (90C40) Optimal stochastic control (93E20) Stochastic learning and adaptive control (93E35)

Cites Work

• Markov chains and stochastic stability
• \({\mathcal Q}\)-learning
• Approximating value functions for controlled degenerate diffusion processes by using piece-wise constant policies.
• On the rate of convergence of finite-difference approximations for Bellman's equations with variable coefficients
• Title not available (Why is that?)
• Title not available (Why is that?)
• On the convergence rate of approximation schemes for Hamilton-Jacobi-Bellman Equations
• Error Bounds for Monotone Approximation Schemes for Hamilton--Jacobi--Bellman Equations
• Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network HJB approach
• An analysis of temporal-difference learning with function approximation
• Algorithms for reinforcement learning.
• Asynchronous stochastic approximation and Q-learning
• Value iteration and adaptive dynamic programming for data-driven adaptive optimal control design
• On the Convergence of Stochastic Iterative Dynamic Programming Algorithms
• Approximate Q Learning for Controlled Diffusion Processes and Its Near Optimality
• Policy gradient in continuous time
• Q-learning for continuous-time linear systems: A model-free infinite horizon optimal control approach
• Title not available (Why is that?)
• Variational estimation of the drift for stochastic differential equations from the empirical density
• Title not available (Why is that?)
• Policy iterations for reinforcement learning problems in continuous time and space -- fundamental theory and methods
• Neural networks-based backward scheme for fully nonlinear PDEs
• Continuous‐time mean–variance portfolio selection: A reinforcement learning framework
• Improved order 1/4 convergence for piecewise constant policy approximation of stochastic control problems
• Sample Complexity of Asynchronous Q-Learning: Sharper Analysis and Variance Reduction
• Title not available (Why is that?)

Cited In (10)

• Data-driven approximate Q-learning stabilization with optimality error bound analysis
• A generalization error for Q-learning
• Optimal learning with \textit{Q}-aggregation
• Title not available (Why is that?)
• Temporal difference-based policy iteration for optimal control of stochastic systems
• Reinforcement Learning for Linear-Convex Models with Jumps via Stability Analysis of Feedback Controls
• Approximate Q Learning for Controlled Diffusion Processes and Its Near Optimality
• Minimax Q-learning control for linear systems using the Wasserstein metric
• Two Time-Scale Stochastic Approximation with Controlled Markov Noise and Off-Policy Temporal-Difference Learning
• Continuity of cost in Borkar control topology and implications on discrete space and time approximations for controlled diffusions under several criteria

This page was built for publication: Approximate Q Learning for Controlled Diffusion Processes and Its Near Optimality

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6136230)