Markov decision processes with incomplete information and semiuniform Feller transition probabilities

DOI10.1137/21M1442152zbMATH Open1498.90245arXiv2108.09232OpenAlexW3164777109WikidataQ113779024 ScholiaQ113779024MaRDI QIDQ5097394FDOQ5097394

Authors: Eugene A. Feinberg, Pavlo O. Kasyanov, Michael Z. Zgurovsky 

Publication date: 23 August 2022

Published in: SIAM Journal on Control and Optimization (Search for Journal in Brave)

Abstract: This paper deals with control of partially observable discrete-time stochastic systems. It introduces and studies Markov Decision Processes with Incomplete Information and with semi-uniform Feller transition probabilities. The important feature of these models is that their classic reduction to Completely Observable Markov Decision Processes with belief states preserves semi-uniform Feller continuity of transition probabilities. Under mild assumptions on cost functions, optimal policies exist, optimality equations hold, and value iterations converge to optimal values for these models. In particular, for Partially Observable Markov Decision Processes the results of this paper imply new and generalize several known sufficient conditions on transition and observation probabilities for weak continuity of transition probabilities for Markov Decision Processes with belief states, the existence of optimal policies, validity of optimality equations defining optimal policies, and convergence of value iterations to optimal values.

Full work available at URL: https://arxiv.org/abs/2108.09232

Recommendations

• Partially observable total-cost Markov decision processes with weakly continuous transition probabilities
• scientific article; zbMATH DE number 4156234
• Average cost Markov decision processes with semi-uniform Feller transition probabilities
• Structural properties for contracting state partially observable Markov decision processes
• scientific article; zbMATH DE number 9756

zbMATH Keywords

Markov decision processincomplete informationoptimality equationsvalue iterationssemiuniform Feller transition probabilities

Mathematics Subject Classification ID

Dynamic programming (90C39) Markov and semi-Markov decision processes (90C40)

Cites Work

• Optimal control of Markov processes with incomplete state information
• Measure theory. Vol. I and II
• State of the Art—A Survey of Partially Observable Markov Decision Processes: Theory, Models, and Algorithms
• The Optimal Control of Partially Observable Markov Processes over the Infinite Horizon: Discounted Costs
• The Optimal Control of Partially Observable Markov Processes over a Finite Horizon
• Title not available (Why is that?)
• Real Analysis and Probability
• Title not available (Why is that?)
• Title not available (Why is that?)
• The Complexity of Markov Decision Processes
• Optimal Infinite-Horizon Undiscounted Control of Finite Probabilistic Systems
• Stochastic optimal control. The discrete time case
• Conditions for optimality in dynamic programming and for the limit of n-stage optimal policies to be optimal
• Adaptive Markov control processes
• Markov decision processes with applications to finance.
• Average Optimality in Dynamic Programming with General State Space
• Title not available (Why is that?)
• On compactness of the space of policies in stochastic dynamic programming
• On dynamic programming: Compactness of the space of policies
• Title not available (Why is that?)
• Title not available (Why is that?)
• Optimal control of partially observable Markovian systems
• Average optimality in dynamic programming on Borel spaces -- unbounded costs and controls
• Average cost Markov decision processes with weakly continuous transition probabilities
• Bayesian dynamic programming
• Average cost Markov decision processes with semi-uniform Feller transition probabilities
• Incomplete information in Markovian decision models
• Partially observable total-cost Markov decision processes with weakly continuous transition probabilities
• Reduction of a Controlled Markov Model with Incomplete Data to a Problem with Complete Information in the Case of Borel State and Control Space
• Convergence of probability measures and Markov decision models with incomplete information
• Title not available (Why is that?)
• Berge's theorem for noncompact image sets
• Fatou's lemma for weakly converging probabilities
• Uniform Fatou's lemma
• On essential information in sequential decision processes
• The critical discount factor for finite Markovian decision processes with an absorbing set
• On convergence of value iteration for a class of total cost Markov decision processes
• MDPs with setwise continuous transition probabilities
• Fatou's lemma in its classical form and Lebesgue's convergence theorems for varying measures with applications to Markov decision processes
• Weak Feller property of non-linear filters

Cited In (12)

• Markov Decision Processes with Imprecise Transition Probabilities
• Partially observable total-cost Markov decision processes with weakly continuous transition probabilities
• Title not available (Why is that?)
• Title not available (Why is that?)
• Structural properties for contracting state partially observable Markov decision processes
• Convergence of probability measures and Markov decision models with incomplete information
• Formalization of methods for the development of autonomous artificial intelligence systems
• Optimality conditions for partially observable Markov decision processes
• Another look at partially observed optimal stochastic control: existence, ergodicity, and approximations without belief-reduction
• Equivalent conditions for weak continuity of nonlinear filters
• Semi-uniform Feller stochastic kernels
• Information-theoretic multi-time-scale partially observable systems with inspiration from leukemia treatment

This page was built for publication: Markov decision processes with incomplete information and semiuniform Feller transition probabilities

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5097394)