Approximate solutions to constrained risk-sensitive Markov decision processes

DOI10.1016/J.EJOR.2023.02.039MaRDI QIDQ6113325zbMATH OpenOpenAlexFDO

Authors Uday M. Kumar, Sanjay P. Bhat, V. Kavitha, Nandyala Hemachandra

Publication date 11 July 2023

Published in European Journal of Operational Research (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2209.14963

zbMATH Keywords

constrained MDPcontrolexponential utilityMarkov decision processes (MDP)risk-sensitive

Mathematics Subject Classification ID

Operations research and management science (90Bxx)

Abstract: This paper considers the problem of finding near-optimal Markovian randomized (MR) policies for finite-state-action, infinite-horizon, constrained risk-sensitive Markov decision processes (CRSMDPs). Constraints are in the form of standard expected discounted cost functions as well as expected risk-sensitive discounted cost functions over finite and infinite horizons. The main contribution is to show that the problem possesses a solution if it is feasible, and to provide two methods for finding an approximate solution in the form of an ultimately stationary (US) MR policy. The latter is achieved through two approximating finite-horizon CRSMDPs which are constructed from the original CRSMDP by time-truncating the original objective and constraint cost functions, and suitably perturbing the constraint upper bounds. The first approximation gives a US policy which is $epsilon$-optimal and feasible for the original problem, while the second approximation gives a near-optimal US policy whose violation of the original constraints is bounded above by a specified $epsilon$. A key step in the proofs is an appropriate choice of a metric that makes the set of infinite-horizon MR policies and the feasible regions of the three CRSMDPs compact, and the objective and constraint functions continuous. A linear-programming-based formulation for solving the approximating finite-horizon CRSMDPs is also given.

Cites work

• A convex analytic approach to risk-aware Markov decision processes
• A multi-product risk-averse newsvendor with exponential utility function
• A Utility Criterion for Markov Decision Processes
• Constrained Discounted Dynamic Programming
• Dynamic programming in constrained Markov decision processes
• scientific article; zbMATH DE number 1348599 (Why is no real title available?)
• scientific article; zbMATH DE number 1461253 (Why is no real title available?)
• scientific article; zbMATH DE number 3793773 (Why is no real title available?)
• scientific article; zbMATH DE number 2243395 (Why is no real title available?)
• Inventory Control with an Exponential Utility Criterion
• Markov decision processes
• Markov decision processes with a new optimality criterion: Discrete time
• Mean-variance analysis of the newsvendor problem with price-dependent, isoelastic demand
• Modeling local coronavirus outbreaks
• More risk-sensitive Markov decision processes
• Probability essentials
• Real analysis
• Risk Aversion in Inventory Management
• Risk-Constrained Markov Decision Processes
• Risk-sensitive and minimax control of discrete-time, finite-state Markov decision processes
• Risk-Sensitive and Risk-Neutral Multiarmed Bandits
• Risk-Sensitive Markov Decision Processes
• Sensitivity analysis and optimal ultimately stationary deterministic policies in some constrained discounted cost models
• Some Remarks on Finite Horizon Markovian Decision Models
• Variance-Penalized Markov Decision Processes
• Variance-penalized Markov decision processes: dynamic programming and reinforcement learning techniques

Cited in

(2)

• On risk-sensitive piecewise deterministic Markov decision processes
• Successive Approximations for Finite Horizon, Semi-Markov Decision Processes with Application to Asset Liquidation

This page was built for publication: Approximate solutions to constrained risk-sensitive Markov decision processes

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6113325)