Infinite Horizon Average Cost Dynamic Programming Subject to Total Variation Distance Ambiguity

DOI10.1137/18M1210514;zbMATH Open1421.93148arXiv1512.06510MaRDI QIDQ5232245FDOQ5232245

Themistoklis Charalambous, I. Tzortzis, Charalambos D. Charalambous

Publication date: 30 August 2019

Published in: SIAM Journal on Control and Optimization (Search for Journal in Brave)

Abstract: We analyze the infinite horizon minimax average cost Markov Control Model (MCM), for a class of controlled process conditional distributions, which belong to a ball, with respect to total variation distance metric, centered at a known nominal controlled conditional distribution with radius $Rin[0,2]$, in which the minimization is over the control strategies and the maximization is over conditional distributions. Upon performing the maximization, a dynamic programming equation is obtained which includes, in addition to the standard terms, the oscillator semi-norm of the cost-to-go. First, the dynamic programming equation is analyzed for finite state and control spaces. We show that if the nominal controlled process distribution is irreducible, then for every stationary Markov control policy the maximizing conditional distribution of the controlled process is also irreducible for $Rin[0,R_{max}]$. Second, the generalized dynamic programming is analyzed for Borel spaces. We derive necessary and sufficient conditions for any control strategy to be optimal. Through our analysis, new dynamic programming equations and new policy iteration algorithms are derived. The main feature of the new policy iteration algorithms (which are applied for finite alphabet spaces) is that the policy evaluation and policy improvement steps are performed by using the maximizing conditional distribution, which is obtained via a water filling solution. Finally, the application of the new dynamic programming equations and the corresponding policy iteration algorithms are shown via illustrative examples.

Full work available at URL: https://arxiv.org/abs/1512.06510

Recommendations

• Dynamic programming subject to total variation distance ambiguity
• Infinite Horizon Stochastic Programs
• Infinite horizon programs; convergence of approximate solutions
• Existence and discovery of average optimal solutions in deterministic infinite horizon optimization
• Average Cost Optimal Stationary Policies in Infinite State Markov Decision Processes with Unbounded Costs
• An infinite-horizon multistage dynamic optimization problem
• Average optimality in nonhomogeneous infinite horizon Markov decision processes
• Infinite horizon stochastic optimal control problems with running maximum cost
• Adaptive aggregation methods for infinite horizon dynamic programming
• Infinite-horizon deterministic dynamic programming in discrete time: a monotone convergence principle and a penalty method

zbMATH Keywords

dynamic programmingminimaxstochastic controltotal variation distancepolicy iterationinfinite horizonaverage costMarkov control models

Mathematics Subject Classification ID

Dynamic programming (90C39) Minimax problems in mathematical programming (90C47) Optimal stochastic control (93E20)

Cites Work

• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• On Choosing and Bounding Probability Metrics
• Title not available (Why is that?)
• Title not available (Why is that?)
• Dynamic programming and stochastic control
• \(H^ \infty\)-optimal control and related minimax design problems. A dynamic game approach.
• Title not available (Why is that?)
• Discrete-Time Controlled Markov Processes with Average Cost Criterion: A Survey
• Minimum principle for partially observable nonlinear risk-sensitive control problems using measure-valued decompositions
• Risk-sensitive control and dynamic games for partially observed discrete-time nonlinear systems
• Distributionally robust Markov decision processes
• A Finite-Dimensional Risk-Sensitive Control Problem
• On Minimum Cost Per Unit Time Control of Markov Chains
• Another set of conditions for average optimality in Markov control processes
• Minimax optimal control of stochastic uncertain systems with relative entropy constraints
• Finite horizon minimax optimal control of stochastic partially observed time varying uncertain systems
• Stochastic Uncertain Systems Subject to Relative Entropy Constraints: Induced Norms and Monotonicity Properties of Minimax Games
• Control of Markov Chains with Long-Run Average Cost Criterion: The Dynamic Programming Equations
• Extremum Problems With Total Variation Distance and Their Applications
• Distributionally Robust Counterpart in Markov Decision Processes
• Robust MDPs with \(k\)-rectangular uncertainty
• Dynamic Programming Subject to Total Variation Distance Ambiguity

Cited In (1)

• Robust Markov Decision Processes with Data-Driven, Distance-Based Ambiguity Sets

This page was built for publication: Infinite Horizon Average Cost Dynamic Programming Subject to Total Variation Distance Ambiguity

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5232245)