Average optimality for continuous-time Markov decision processes in Polish spaces

DOI10.1214/105051606000000105zbMATH Open1160.90010arXivmath/0607098OpenAlexW2026850974MaRDI QIDQ997948FDOQ997948

Xianping Guo, Ulrich Rieder

Publication date: 8 August 2007

Published in: The Annals of Applied Probability (Search for Journal in Brave)

Abstract: This paper is devoted to studying the average optimality in continuous-time Markov decision processes with fairly general state and action spaces. The criterion to be maximized is expected average rewards. The transition rates of underlying continuous-time jump Markov processes are allowed to be unbounded, and the reward rates may have neither upper nor lower bounds. We first provide two optimality inequalities with opposed directions, and also give suitable conditions under which the existence of solutions to the two optimality inequalities is ensured. Then, from the two optimality inequalities we prove the existence of optimal (deterministic) stationary policies by using the Dynkin formula. Moreover, we present a ``semimartingale characterization of an optimal stationary policy. Finally, we use a generalized Potlach process with control to illustrate the difference between our conditions and those in the previous literature, and then further apply our results to average optimal control problems of generalized birth--death systems, upwardly skip-free processes and two queueing systems. The approach developed in this paper is slightly different from the ``optimality inequality approach widely used in the previous literature.

Full work available at URL: https://arxiv.org/abs/math/0607098

zbMATH Keywords

average rewardoptimality inequalityoptimal stationary policygeneral state spacesemimartingale characterization

Mathematics Subject Classification ID

Markov and semi-Markov decision processes (90C40) Optimal stochastic control (93E20)

Cites Work

• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• Bias Optimality in Controlled Queueing Systems
• Finite state continuous time Markov decision processes with an infinite planning horizon
• Stability of Markovian processes III: Foster–Lyapunov criteria for continuous-time processes
• Zero-sum games for continuous-time Markov chains with unbounded transition and average payoff rates
• Average Optimality in Markov Control Processes via Discounted-Cost Problems and Linear Programming
• Continuous-time Markov chains. An applications-oriented approach
• Equivalence of exponential ergodicity and \(L^ 2\)-exponential convergence for Markov chains.
• Exponential and uniform ergodicity of Markov processes
• Computable exponential convergence rates for stochastically ordered Markov processes
• Drift and monotonicity conditions for continuous-time controlled markov chains with an average criterion
• Some estimates of the rate of convergence for birth and death processes
• Generalized potlatch and smoothing processes
• Optimal Control of Ergodic Continuous-Time Markov Chains with Average Sample-Path Rewards
• Criteria for ergodicity, exponential ergodicity and strong ergodicity of Markov processes
• Measurable selection theorems for optimization problems
• Continuous-time controlled Markov chains.
• On the Integro-Differential Equations of Purely Discontinuous Markoff Processes
• On maximal rewards and \(\varepsilon\)-optimal policies in continuous time Markov decision chains
• A note on bias optimality in controlled queueing systems
• A note on optimality conditions for continuous-time Markov decision processes with average cost criterion
• Nondiscounted Continuous Time Markovian Decision Process with Countable State Space
• Continuous time control of Markov processes on an arbitrary state space: Discounted rewards
• On Homogeneous Markov Models with Continuous Time and Finite or Countable State Space
• Continuous time Markov decision programming with average reward criterion and unbounded reward rate

Cited In (30)

• Estimate the exponential convergence rate of \(f\)-ergodicity via spectral gap
• Absorbing continuous-time Markov decision processes with total cost criteria
• Title not available (Why is that?)
• Average optimality for continuous-time Markov decision processes under weak continuity conditions
• Nonzero-sum games for continuous-time jump processes under the expected average payoff criterion
• A survey of recent results on continuous-time Markov decision processes (with comments and rejoinder)
• Risk sensitive control of pure jump processes on a general state space
• New average optimality conditions for semi-Markov decision processes in Borel spaces
• Bias and Overtaking Optimality for Continuous-Time Jump Markov Decision Processes in Polish Spaces
• New sufficient conditions for average optimality in continuous-time Markov decision processes
• Construction and regularity of transition functions on Polish spaces under measurability conditions
• A note on the existence of optimal stationary policies for average Markov decision processes with countable states
• The policy iteration algorithm for average continuous control of piecewise deterministic Markov processes
• Continuous-time Markov decision processes with \(n\)th-bias optimality criteria
• The Vanishing Discount Approach for the Average Continuous Control of Piecewise Deterministic Markov Processes
• Optimality of mixed policies for average continuous-time Markov decision processes with constraints
• Discounted continuous-time constrained Markov decision processes in Polish spaces
• Uniform ergodicity of continuous-time controlled Markov chains: a survey and new results
• Average stochastic games for continuous-time jump processes
• Kolmogorov's Equations for Jump Markov Processes and Their Applications to Control Problems
• Average optimality for continuous-time Markov decision processes with a policy iteration approach
• Average optimality inequality for continuous-time Markov decision processes in Polish spaces
• Verifiable conditions for average optimality of continuous-time Markov decision processes
• Average sample-path optimality for continuous-time Markov decision processes in Polish spaces
• On solutions of Kolmogorov's equations for nonhomogeneous jump Markov processes
• Computable approximations for average Markov decision processes in continuous time
• New discount and average optimality conditions for continuous-time Markov decision processes
• Policy iteration for continuous-time average reward Markov decision processes in Polish spaces
• Variance minimization for continuous-time Markov decision processes: two approaches
• Strong \(n\)-discount and finite-horizon optimality for continuous-time Markov decision processes

This page was built for publication: Average optimality for continuous-time Markov decision processes in Polish spaces

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q997948)