Proximal policy optimization learning based control of congested freeway traffic

DOI10.1002/OCA.3068MaRDI QIDQ6125666zbMATH OpenOpenAlexFDO

Authors Nailong Wu, Jie Qi, Anqi Pan, Zhiguang Feng, Huaicheng Yan, Yueying Wang

Publication date 11 April 2024

Published in Optimal Control Applications \& Methods (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2204.05627

zbMATH Keywords

adaptive controlinput delaytraffic flowadaptive cruise controlproximal policy optimization

Mathematics Subject Classification ID

Systems theory; control (93-XX)

Abstract: This study proposes a delay-compensated feedback controller based on proximal policy optimization (PPO) reinforcement learning to stabilize traffic flow in the congested regime by manipulating the time-gap of adaptive cruise control-equipped (ACC-equipped) vehicles.The traffic dynamics on a freeway segment are governed by an Aw-Rascle-Zhang (ARZ) model, consisting of $2imes2$ nonlinear first-order partial differential equations (PDEs).Inspired by the backstepping delay compensator [18] but different from whose complex segmented control scheme, the PPO control is composed of three feedbacks, namely the current traffic flow velocity, the current traffic flow density and previous one step control input. The control gains for the three feedbacks are learned from the interaction between the PPO and the numerical simulator of the traffic system without knowing the system dynamics. Numerical simulation experiments are designed to compare the Lyapunov control, the backstepping control and the PPO control. The results show that for a delay-free system, the PPO control has faster convergence rate and less control effort than the Lyapunov control. For a traffic system with input delay, the performance of the PPO controller is comparable to that of the Backstepping controller, even for the situation that the delay value does not match. However, the PPO is robust to parameter perturbations, while the Backstepping controller cannot stabilize a system where one of the parameters is disturbed by Gaussian noise.

Cites work

• Capacity drop and traffic control for a second order traffic model
• Comparative model accuracy of a data-fitted generalized Aw-Rascle-Zhang model
• Delay-Compensated Distributed PDE Control of Traffic With Connected/Automated Vehicles
• Delay-Robust Control Design for Two Heterodirectional Linear Coupled Hyperbolic PDEs
• Derivation of second order traffic flow models with time delays
• Feedback Control of Nonlinear Hyperbolic PDE Systems Inspired by Traffic Flow Models
• Instability of cooperative adaptive cruise control traffic flow: a macroscopic approach
• Macroscopic traffic flow modeling with adaptive cruise control: development and numerical solution
• Necessary and Sufficient Conditions on the Exponential Stability of Positive Hyperbolic Systems
• On kinematic waves II. A theory of traffic flow on long crowded roads
• Output feedback control of two-lane traffic congestion
• Prediction of traffic convective instability with spectral analysis of the Aaw-Rascle-Zhang model
• Resurrection of "Second Order" Models of Traffic Flow
• Shock waves on the highway
• Traffic congestion control for Aw-Rascle-Zhang model

Cited in

(2)

• Compact formulation of the augmented evolution equation for optimal control computation
• RBF neural network-based adaptive control for Aw-Rascle-Zhang mixed traffic flow model with unknown external disturbances

This page was built for publication: Proximal policy optimization learning based control of congested freeway traffic

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6125666)