Average cost Markov decision processes with weakly continuous transition probabilities

DOI10.1287/MOOR.1120.0555zbMATH Open1297.90173arXiv1202.4122OpenAlexW1966208686MaRDI QIDQ2925348FDOQ2925348

Authors: Eugene A. Feinberg, Pavlo O. Kasyanov, Nina V. Zadoianchuk 

Publication date: 21 October 2014

Published in: Mathematics of Operations Research (Search for Journal in Brave)

Abstract: This paper presents sufficient conditions for the existence of stationary optimal policies for average-cost Markov Decision Processes with Borel state and action sets and with weakly continuous transition probabilities. The one-step cost functions may be unbounded, and action sets may be noncompact. The main contributions of this paper are: (i) general sufficient conditions for the existence of stationary discount-optimal and average-cost optimal policies and descriptions of properties of value functions and sets of optimal actions, (ii) a sufficient condition for the average-cost optimality of a stationary policy in the form of optimality inequalities, and (iii) approximations of average-cost optimal actions by discount-optimal actions.

Full work available at URL: https://arxiv.org/abs/1202.4122

Recommendations

• Average cost Markov decision processes with semi-uniform Feller transition probabilities
• Average cost Markov decision processes: Optimality conditions
• Average optimality for continuous-time Markov decision processes under weak continuity conditions
• Average cost Markov decision processes under the hypothesis of Doeblin
• Partially observable total-cost Markov decision processes with weakly continuous transition probabilities
• Continuous-time Markov decision processes under the risk-sensitive average cost criterion
• Denumerable continuous-time Markov decision processes with multiconstraints on average costs
• Average Reward Markov Decision Processes with Multiple Cost Constraints
• Markov Decision Processes with a Borel Measurable Cost Function—The Average Case

zbMATH Keywords

Markov decision processoptimal policyoptimality inequalityaverage cost per unit time

Mathematics Subject Classification ID

Dynamic programming (90C39) Markov and semi-Markov decision processes (90C40)

Cites Work

• On sequential decisions and Markov chains
• Discrete Dynamic Programming
• Average Optimality in Dynamic Programming with General State Space
• Markovian Sequential Replacement Processes
• Fatou's lemma and Lebesgue's convergence theorem for measures
• Optimal decision procedures for finite markov chains. Part I: Examples
• A counterexample on the optimality equation in Markov decision chains with the average cost criterion
• Average optimality in dynamic programming on Borel spaces -- unbounded costs and controls
• Title not available (Why is that?)
• OPTIMALITY OF FOUR-THRESHOLD POLICIES IN INVENTORY SYSTEMS WITH CUSTOMER RETURNS AND BORROWING/STORAGE OPTIONS
• Optimality Inequalities for Average Cost Markov Decision Processes and the Stochastic Cash Balance Problem
• Fatou's lemma for weakly converging probabilities
• Non-Discounted Denumerable Markovian Decision Models
• Arbitrary State Markovian Decision Processes
• A Counterexample on the Semicontinuity of Minima
• On the Nonexistence of $|varepsilon$-Optimal Randomized Stationary Policies in Average Cost Markov Decision Models
• Compactness of the space of non-randomized policies in countable-state sequential decision processes

Cited In (51)

• Uniform Fatou's lemma
• Markov decision processes with incomplete information and semiuniform Feller transition probabilities
• Fatou's lemma for weakly converging measures under the uniform integrability condition
• The average cost of Markov chains subject to total variation distance uncertainty
• A mixed value and policy iteration method for stochastic control with universally measurable policies
• Berge's maximum theorem for noncompact image sets
• The Existence of a Minimum Pair of State and Policy for Markov Decision Processes under the Hypothesis of Doeblin
• Average optimality for continuous-time Markov decision processes under weak continuity conditions
• LP based upper and lower bounds for Cesàro and Abel limits of the optimal values in problems of control of stochastic discrete time systems
• Average Optimality in Dynamic Programming with General State Space
• Title not available (Why is that?)
• On the vanishing discount factor approach for Markov decision processes with weakly continuous transition probabilities
• Examples concerning Abel and Cesàro limits
• Partially observable total-cost Markov decision processes with weakly continuous transition probabilities
• On the reduction of total-cost and average-cost MDPs to discounted mdps
• Stochastic setup-cost inventory model with backorders and quasiconvex cost functions
• Unbounded dynamic programming via the Q-transform
• On the Minimum Pair Approach for Average Cost Markov Decision Processes with Countable Discrete Action Spaces and Strictly Unbounded Costs
• Planning for the long run: programming with patient, Pareto responsive preferences
• Approximation of average cost Markov decision processes using empirical distributions and concentration inequalities
• Solutions of the average cost optimality equation for Markov decision processes with weakly continuous kernel: the fixed-point approach revisited
• A note on the existence of optimal stationary policies for average Markov decision processes with countable states
• Reduction of total-cost and average-cost MDPs with weakly continuous transition probabilities to discounted mdps
• A survey of average cost problems in deterministic discrete-time control systems
• On convergence of value iteration for a class of total cost Markov decision processes
• Near optimality of quantized policies in stochastic control under weak continuity conditions
• A useful technique for piecewise deterministic Markov decision processes
• Constrained Markov decision processes in Borel spaces: from discounted to average optimality
• Continuity of discounted values and the structure of optimal policies for <scp>periodic‐review</scp> inventory systems with setup costs
• Average cost Markov decision processes: Optimality conditions
• Functional characterization for average cost Markov decision processes with Doeblin's conditions
• Fatou's lemma in its classical form and Lebesgue's convergence theorems for varying measures with applications to Markov decision processes
• Another set of conditions for Markov decision processes with average sample-path costs
• MDPs with setwise continuous transition probabilities
• Average cost Markov decision processes with semi-uniform Feller transition probabilities
• Convergence of probability measures and Markov decision models with incomplete information
• Continuity of minima: local results
• On strong average optimality of Markov decision processes with unbounded costs
• Formalization of methods for the development of autonomous artificial intelligence systems
• Optimality conditions for partially observable Markov decision processes
• Berge's theorem for noncompact image sets
• Title not available (Why is that?)
• New discount and average optimality conditions for continuous-time Markov decision processes
• Another look at partially observed optimal stochastic control: existence, ergodicity, and approximations without belief-reduction
• Convex analytic method revisited: further optimality results and performance of deterministic policies in average cost stochastic control
• Average cost optimality inequality for Markov decision processes with Borel spaces and universally measurable policies
• Title not available (Why is that?)
• On structural properties of optimal average cost functions in Markov decision processes with Borel spaces and universally measurable policies
• Continuity of equilibria for two-person zero-sum games with noncompact action sets and unbounded payoffs
• On the optimality equation for average cost Markov decision processes and its validity for inventory control
• Structure of optimal policies to periodic-review inventory models with convex costs and backorders for all values of discount factors

This page was built for publication: Average cost Markov decision processes with weakly continuous transition probabilities

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2925348)