Undiscounted control policy generation for continuous-valued optimal control by approximate dynamic programming

DOI10.1080/00207179.2021.1939892zbMATH Open1500.49015arXiv2104.11093OpenAlexW3169251661MaRDI QIDQ5043547FDOQ5043547

Authors: Jonathan Lock, T. McKelvey 

Publication date: 6 October 2022

Published in: International Journal of Control (Search for Journal in Brave)

Abstract: We present a numerical method for generating the state-feedback control policy associated with general undiscounted, constant-setpoint, infinite-horizon, nonlinear optimal control problems with continuous state variables. The method is based on approximate dynamic programming, and is closely related to approximate policy iteration. Existing methods typically terminate based on the convergence of the control policy and either require a discounted problem formulation or demand the cost function to lie in a specific subclass of functions. The presented method extends on existing termination criteria by requiring both the control policy and the resulting system state to converge, allowing for use with undiscounted cost functions that are bounded and continuous. This paper defines the numerical method, derives the relevant underlying mathematical properties, and validates the numerical method with representative examples. A MATLAB implementation with the shown examples is freely available.

Full work available at URL: https://arxiv.org/abs/2104.11093

Recommendations

• Temporal difference-based policy iteration for optimal control of stochastic systems
• Controller design and value function approximation for nonlinear dynamical systems
• Optimal control for nonlinear continuous systems by adaptive dynamic programming based on fuzzy basis functions
• Continuous-action planning for discounted infinite-horizon nonlinear optimal control with Lipschitz values
• Continuous state dynamic programming via nonexpansive approximation

zbMATH Keywords

optimal controlapproximate dynamic programmingcontrol policyundiscounted infinite-horizon

Mathematics Subject Classification ID

Approximation methods and heuristics in mathematical programming (90C59) Dynamic programming (90C39) Dynamic programming in optimal control and differential games (49L20)

Cites Work

• Title not available (Why is that?)
• Turnpike phenomenon and infinite horizon optimal control
• The turnpike property in finite-dimensional nonlinear optimal control
• Approximate policy iteration: a survey and some new methods
• Dynamic programming and optimal control. Vol. 2
• Variable resolution discretization in optimal control
• What you should know about approximate dynamic programming
• The theory of dynamic programming
• Continuous state dynamic programming via nonexpansive approximation
• On the Convergence of Policy Iteration in Stationary Dynamic Programming
• Analysis of a Numerical Dynamic Programming Algorithm Applied to Economic Models
• Existence of optimal stationary policies in deterministic optimal control
• Convergence Properties of Policy Iteration
• Dynamic programming and optimal control. Vol. 1.

Cited In (2)

• Controller design and value function approximation for nonlinear dynamical systems
• Temporal difference-based policy iteration for optimal control of stochastic systems

This page was built for publication: Undiscounted control policy generation for continuous-valued optimal control by approximate dynamic programming

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5043547)