Undiscounted control policy generation for continuous-valued optimal control by approximate dynamic programming

DOI10.1080/00207179.2021.1939892MaRDI QIDQ5043547zbMATH OpenOpenAlexFDO

Authors Jonathan Lock, T. McKelvey

Publication date 6 October 2022

Published in International Journal of Control (Search for Journal in Brave)

Copyright license Creative Commons Attribution 4.0 International

Full work available at URL https://arxiv.org/abs/2104.11093

zbMATH Keywords

optimal controlapproximate dynamic programmingcontrol policyundiscounted infinite-horizon

Mathematics Subject Classification ID

Approximation methods and heuristics in mathematical programming (90C59) Dynamic programming (90C39) Dynamic programming in optimal control and differential games (49L20)

Abstract: We present a numerical method for generating the state-feedback control policy associated with general undiscounted, constant-setpoint, infinite-horizon, nonlinear optimal control problems with continuous state variables. The method is based on approximate dynamic programming, and is closely related to approximate policy iteration. Existing methods typically terminate based on the convergence of the control policy and either require a discounted problem formulation or demand the cost function to lie in a specific subclass of functions. The presented method extends on existing termination criteria by requiring both the control policy and the resulting system state to converge, allowing for use with undiscounted cost functions that are bounded and continuous. This paper defines the numerical method, derives the relevant underlying mathematical properties, and validates the numerical method with representative examples. A MATLAB implementation with the shown examples is freely available.

Recommendations

• Temporal difference-based policy iteration for optimal control of stochastic systems
• Controller design and value function approximation for nonlinear dynamical systems
• Optimal control for nonlinear continuous systems by adaptive dynamic programming based on fuzzy basis functions
• Continuous-action planning for discounted infinite-horizon nonlinear optimal control with Lipschitz values
• Continuous state dynamic programming via nonexpansive approximation

Cites work

• scientific article; zbMATH DE number 700091 (Why is no real title available?)
• Analysis of a Numerical Dynamic Programming Algorithm Applied to Economic Models
• Approximate policy iteration: a survey and some new methods
• Continuous state dynamic programming via nonexpansive approximation
• Convergence Properties of Policy Iteration
• Dynamic programming and optimal control. Vol. 1.
• Dynamic programming and optimal control. Vol. 2
• Existence of optimal stationary policies in deterministic optimal control
• On the Convergence of Policy Iteration in Stationary Dynamic Programming
• The theory of dynamic programming
• The turnpike property in finite-dimensional nonlinear optimal control
• Turnpike phenomenon and infinite horizon optimal control
• Variable resolution discretization in optimal control
• What you should know about approximate dynamic programming

Cited in

(2)

• Controller design and value function approximation for nonlinear dynamical systems
• Temporal difference-based policy iteration for optimal control of stochastic systems

This page was built for publication: Undiscounted control policy generation for continuous-valued optimal control by approximate dynamic programming

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5043547)