A Stochastic Composite Augmented Lagrangian Method for Reinforcement Learning
From MaRDI portal
Abstract: In this paper, we consider the linear programming (LP) formulation for deep reinforcement learning. The number of the constraints depends on the size of state and action spaces, which makes the problem intractable in large or continuous environments. The general augmented Lagrangian method suffers the double-sampling obstacle in solving the LP. Namely, the conditional expectations originated from the constraint functions and the quadratic penalties in the augmented Lagrangian function impose difficulties in sampling and evaluation. Motivated from the updates of the multipliers, we overcome the obstacles in minimizing the augmented Lagrangian function by replacing the intractable conditional expectations with the multipliers. Therefore, a deep parameterized augment Lagrangian method is proposed. Furthermore, the replacement provides a promising breakthrough to integrate the two steps in the augmented Lagrangian method into a single constrained problem. A general theoretical analysis shows that the solutions generated from a sequence of the constrained optimizations converge to the optimal solution of the LP if the error is controlled properly. A theoretical analysis on the quadratic penalty algorithm under neural tangent kernel setting shows the residual can be arbitrarily small if the parameter in network and optimization algorithm is chosen suitably. Preliminary experiments illustrate that our method is competitive to other state-of-the-art algorithms.
Recommendations
- Data-driven optimal control with a relaxed linear program
- A unified DC programming framework and efficient DCA based approaches for large scale batch reinforcement learning
- A stochastic trust-region framework for policy optimization
- Deep neural networks algorithms for stochastic control problems on finite horizon: convergence analysis
- Accelerating Stochastic Composition Optimization
Cites work
- Convex analysis and monotone operator theory in Hilbert spaces
- scientific article; zbMATH DE number 700091 (Why is no real title available?)
- scientific article; zbMATH DE number 1095138 (Why is no real title available?)
- Monotone Operators and the Proximal Point Algorithm
- Randomized linear programming solves the Markov decision problem in nearly linear (sometimes sublinear) time
- Some continuity properties of polyhedral multifunctions
- Stochastic compositional gradient descent: algorithms for minimizing compositions of expected-value functions
This page was built for publication: A Stochastic Composite Augmented Lagrangian Method for Reinforcement Learning
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6161305)
