Sensitivity of constrained Markov decision processes
The paper considers a stationary parameter, discrete time, finite state, finite action Markov decision process, where \(X_ t\in X\), \(A_ t\in A\) are, respectively, the random state and action at time \(t\); \(x\) is the initial state; \(\beta\), \(0<\beta\leq 1\), is a discount factor; \(S\) is the set of stationary Markov policies; \(c(y,a)\), \(\{d^ k(y,a)\}\), \(1\leq k\leq K\), are cost functions; and \(\{V_ k\}\) are preset constraint levels. If \(E^ u_ x\) is the expectation operator, given the initial state \(x\) and policy \(u\in S\), the main problem addressed is \(\text{COP}_ \beta(x)\) \[ \text{minimize }\biggl[C_ \beta(x,u):=(1-\beta)E^ u_ x\Bigl[\sum^ \infty_{s=0}\beta^ sc(X_ s,A_ s)\Bigl]\biggl] \] \[ \text{subject to }D^ k_ \beta:=(1-\beta)E^ u_ x\Bigl[\sum^ \infty_{s=0}\rho^ sd^ k(X_ s,A_ s)\Bigl]\leq V_ k, \quad 1\leq k\leq K,\;u\in S. \] The associated linear program is, with \(\{P_{yav}\}\) being the transition probabilities and \(\delta_ v(y)\) being the Kronecker function, \(\text{LP}_ \beta(x)\) \[ \text{minimize }\Bigl[C(z):=\sum_{y,a}c(y,a)z(y,a)\Bigl] \] \[ \text{subject to }\sum_{y,a}z(y,a)(\delta_ v(y)-\beta P_{yav})=(1-\beta)\delta_ x(v),\quad v\in X, \] \[ \text{and }D^ k(z):=\sum_{y,a}d^ k(y,a)z(y,a)\leq V_ k,\quad 1\leq k\leq K. \] Under a positive recurrent state assumption, it is, in effect, shown that \(\text{COP}_ \beta(x)\) and \(\text{LP}_ \beta(x)\) are equivalent problems. The main purpose of the paper is to study the continuity properties of \(\text{COP}_ \beta(x)\) in terms of the parameters \(\{\beta, P_{yav}, c(y,a), \{d^ k(y,a)\}\}\). These are replaced by sequences \(\{\beta_ n, P^ n_{yav}, c_ n(y,a),\allowbreak \{d^ k_ n(y,a)\}\}\) with specified convergence properties. \(\text{LP}_ \beta(x)\) is then generalized to \(\{\text{LP}^ n_ \beta(x)\}\) and it is shown that various limiting properties of \(\{\text{LP}^ n_ \beta\}\) hold in relationship to \(\text{LP}_ \beta\). This is used to establish continuity results. -- Some consideration to limiting finite horizon problems and to adaptive problems is given.
- scientific article; zbMATH DE number 3987072
- Sensitivity analysis in Markov decision processes with uncertain reward parameters
- scientific article; zbMATH DE number 2159039
- Sensitivity analysis in discounted Markovian decision problems
- Optimality equations and sensitive optimality in bounded Markov decision processes1
- On constrained Markov decision processes
- scientific article; zbMATH DE number 1348599
- A sensitivity-based construction approach to variance minimization of Markov decision processes
- A convex analytic approach to Markov decision processes
- Adaptive control of constrained Markov chains
- Adaptive control of constrained Markov chains: Criteria and policies
- Constrained Undiscounted Stochastic Dynamic Programming
- Controlled Markov chains with constraints.
- Estimation and control in discounted stochastic dynamic programming
- Finite state Markovian decision processes
- scientific article; zbMATH DE number 3928750 (Why is no real title available?)
- scientific article; zbMATH DE number 3956782 (Why is no real title available?)
- scientific article; zbMATH DE number 3793773 (Why is no real title available?)
- Linear programming and sequential decisions
- Markov Decision Problems and State-Action Frequencies
- On the continuity of the minimum set of a continuous function
- Optimal priority assignment with hard constraint
- Optimal scheduling of interactive and noninteractive traffic in telecommunication systems
- Randomized and Past-Dependent Policies for Markov Decision Processes with Multiple Constraints
- Solving stochastic dynamic programming problems by linear programming — An annotated bibliography
- Some Remarks on Finite Horizon Markovian Decision Models
- Sensitivity analysis of a sequential decision problem with learning
- Continuity of the value of competitive Markov decision processes
- First-order sensitivity of the optimal value in a Markov decision model with respect to deviations in the transition probability function
- Non-randomized policies for constrained Markov decision processes
- scientific article; zbMATH DE number 3970555 (Why is no real title available?)
- scientific article; zbMATH DE number 3987072 (Why is no real title available?)
- Monotonicity in a Markov Decision Process
- Discounted Cost Markov Decision Processes with a Constraint
- Constrained Semi-Markov decision processes with average rewards
- Sensitivity analysis for Markov reward structures until entrance times
- Asymptotic properties of constrained Markov Decision Processes
- Joint chance-constrained Markov decision processes
- On the continuity of the projection mapping from strategic measures to occupation measures in absorbing Markov decision processes
- Lipschitz continuity of value functions in Markovian decision processes
- Constrained cost-coupled stochastic games with independent state processes
- Saddle-point calculation for constrained finite Markov chains
- Sensitivity analysis and optimal ultimately stationary deterministic policies in some constrained discounted cost models
This page was built for publication: Sensitivity of constrained Markov decision processes
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q1176864)
