The variational calculus and approximation in policy space for Markovian decision processes
The functional equations of Markovian decision processes yield the state values (and gain rate in the undiscounted case). Variational expressions are exhibited here for these state values (and gain rate); these expressions are stationary when evaluated at the correct values. When guesses for the values (and gain rate) are inserted into these variational expressions, a superior guess is usually obtained. Repetition of this procedure is shown to be equivalent to the method of successive approximations in policy space. Two other unusual features of this procedure are these: when the linear equations determining the Lagrange multipliers are non-singular, the variational expressions for the state variables are precisely one Newton-Raphson iteration; when applied to a linear objective function and piecewise-linear constraints, which arises for the functional equations of Markovian decision processes, the variational test quantity is piecewise constant, i.e., its first variation and higher variations all vanish. The latter explains its good performance (one-step convergence) if good estimates are available.
- Variational characterizations in Markov decision processes
- scientific article; zbMATH DE number 2159172
- Variance minimization of parameterized Markov decision processes
- Markov Decision Processes with Variance Minimization: A New Condition and Approach
- Optimization of Markov decision processes under the variance criterion
- Variance minimization for continuous-time Markov decision processes: two approaches
- On Finding Optimal Policies for Markov Decision Chains: A Unifying Framework for Mean-Variance-Tradeoffs
- scientific article; zbMATH DE number 4037634
- Computational approaches to variance-penalized Markov decision processes
- Generalized polynomial approximations in Markovian decision processes
- Algorithms for Stochastic Games with Geometrical Interpretation
- Discrete Dynamic Programming with Sensitive Discount Optimality Criteria
- scientific article; zbMATH DE number 3718878 (Why is no real title available?)
- scientific article; zbMATH DE number 3208653 (Why is no real title available?)
- scientific article; zbMATH DE number 3276736 (Why is no real title available?)
- scientific article; zbMATH DE number 3381785 (Why is no real title available?)
- scientific article; zbMATH DE number 3085434 (Why is no real title available?)
- Markov Renewal Programs with Small Interest Rates
- Markov-Renewal Programming. I: Formulation, Finite Return Models
- On Finding Optimal Policies in Discrete Dynamic Programming with No Discounting
- On the Convergence of Policy Iteration in Stationary Dynamic Programming
- Perturbation theory and finite Markov chains
- Perturbation Theory and Undiscounted Markov Renewal Programming
- The Functional Equations of Undiscounted Markov Renewal Programming
- Generalized polynomial approximations in Markovian decision processes
- Variational characterizations in Markov decision processes
- Successive Approximation Methods for Solving Nested Functional Equations in Markov Decision Problems
- Technical note -- cyclic variables and Markov decision processes
- Policy Improvement and the Newton-Raphson Algorithm
- Technical Note—Successive Approximations in Value Determination for a Markov Decision Process
This page was built for publication: The variational calculus and approximation in policy space for Markovian decision processes
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q1068009)
