Complete stability analysis of a heuristic approximate dynamic programming control design

DOI10.1016/J.AUTOMATICA.2015.06.001zbMATH Open1338.90442arXiv1308.3282OpenAlexW173093232MaRDI QIDQ894322FDOQ894322

Authors: Yury Sokolov, Robert Kozma, Ludmilla D. Werbos, Paul J. Werbos 

Publication date: 30 November 2015

Published in: Automatica (Search for Journal in Brave)

Abstract: This paper provides new stability results for Action-Dependent Heuristic Dynamic Programming (ADHDP), using a control algorithm that iteratively improves an internal model of the external world in the autonomous system based on its continuous interaction with the environment. We extend previous results by ADHDP control to the case of general multi-layer neural networks with deep learning across all layers. In particular, we show that the introduced control approach is uniformly ultimately bounded (UUB) under specific conditions on the learning rates, without explicit constraints on the temporal discount factor. We demonstrate the benefit of our results to the control of linear and nonlinear systems, including the cart-pole balancing problem. Our results show significantly improved learning and control performance as compared to the state-of-art.

Full work available at URL: https://arxiv.org/abs/1308.3282

Recommendations

• A boundedness result for the direct heuristic dynamic programming
• Robust reinforcement learning control with static and dynamic stability
• Adaptive dynamic programming for control. Algorithms and stability
• Performance evaluation of direct heuristic dynamic programming using control-theoretic measures
• Convergence analysis of the deep neural networks based globalized dual heuristic programming

zbMATH Keywords

adaptive controlLyapunov functionneural networkadaptive dynamic programminggradient descentadaptive criticaction-dependent heuristic dynamic programming

Mathematics Subject Classification ID

Approximation methods and heuristics in mathematical programming (90C59) Dynamic programming (90C39)

Cites Work

• Universal approximation bounds for superpositions of a sigmoidal function
• Title not available (Why is that?)
• Stability of dynamical systems. Continuous, discontinuous, and discrete systems
• Approximate dynamic programming. Solving the curses of dimensionality
• Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network HJB approach
• Neural network control of nonlinear discrete-time systems.
• Adaptive dynamic programming for control. Algorithms and stability
• Optimal control of unknown nonaffine nonlinear discrete-time systems based on adaptive dynamic programming
• An iterative adaptive dynamic programming method for solving a class of nonlinear zero-sum differential games
• Model-free \(Q\)-learning designs for linear discrete-time zero-sum games with application to \(H^\infty\) control
• A boundedness result for the direct heuristic dynamic programming
• A nearly optimal control scheme using adaptive dynamic programming based on the generalized fuzzy hyperbolic model

Cited In (26)

• Convergence analysis of the deep neural networks based globalized dual heuristic programming
• Spacecraft output feedback attitude control based on extended state observer and adaptive dynamic programming
• A Q-learning predictive control scheme with guaranteed stability
• Data-driven adaptive dynamic programming for partially observable nonzero-sum games via \(Q\)-learning method
• Q-learning solution for optimal consensus control of discrete-time multiagent systems using reinforcement learning
• Data-driven approximate Q-learning stabilization with optimality error bound analysis
• Three bounded proofs for nonlinear multi-input multi-output approximate dynamic programming based on the Lyapunov stability theory
• A boundedness result for the direct heuristic dynamic programming
• Application of heuristic programming to dynamic system stabilization
• Neural-network-based discounted optimal control via an integrated value iteration with accuracy guarantee
• Heuristic dynamic programming-based learning control for discrete-time disturbed multi-agent systems
• Model‐free optimal tracking over finite horizon using adaptive dynamic programming
• Dynamic event-triggered control for discrete-time nonlinear Markov jump systems using policy iteration-based adaptive dynamic programming
• Performance evaluation of direct heuristic dynamic programming using control-theoretic measures
• Event-triggered adaptive dynamic programming for discrete-time multi-player games
• Toward reliable designs of data-driven reinforcement learning tracking control for Euler-Lagrange systems
• Optimal consensus control for double‐integrator multiagent systems with unknown dynamics using adaptive dynamic programming
• Adaptive event-triggered control and observer design for discrete-time nonlinear Markov jump systems with DoS attacks using policy iteration-based adaptive dynamic programming
• Model‐free incremental adaptive dynamic programming based approximate robust optimal regulation
• A stabilizing reinforcement learning approach for sampled systems with partially unknown models
• Neural‐network‐based control for discrete‐time nonlinear systems with denial‐of‐service attack: The adaptive event‐triggered case
• Optimal reconstruction of constrained Janbu method with ADP and non-integral safety factor
• Prioritized experience replay based reinforcement learning for adaptive tracking control of autonomous underwater vehicle
• Robust reinforcement learning control with static and dynamic stability
• Data-driven approximate value iteration with optimality error bound analysis
• Attitude control with auxiliary structure based on adaptive dynamic programming for reentry vehicles

This page was built for publication: Complete stability analysis of a heuristic approximate dynamic programming control design

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q894322)