Stochastic Approximation for Risk-aware Markov Decision Processes

DOI10.1109/TAC.2020.2989702MaRDI QIDQ6301452arXivFDO

Authors Wenjie Huang, William B. Haskell

Publication date 11 May 2018

Published in arXiv

Full work available at URL https://doi.org/10.1109/TAC.2020.2989702

Mathematics Subject Classification ID

Approximation algorithms (68W25) Markov and semi-Markov decision processes (90C40) Risk models (general) (91B05)

arXiv classification

cs.AImath.OC

Abstract: We develop a stochastic approximation-type algorithm to solve finite state/action, infinite-horizon, risk-aware Markov decision processes. Our algorithm has two loops. The inner loop computes the risk by solving a stochastic saddle-point problem. The outer loop performs $Q$-learning to compute an optimal risk-aware policy. Several widely investigated risk measures (e.g. conditional value-at-risk, optimized certainty equivalent, and absolute semi-deviation) are covered by our algorithm. Almost sure convergence and the convergence rate of the algorithm are established. For an error tolerance $epsilon>0$ for the optimal $Q$-value estimation gap and learning rate $kin(1/2,,1]$, the overall convergence rate of our algorithm is $Omega((ln(1/deltaepsilon)/epsilo{n}^2{)}^{1/k}+(ln(1/epsilon){)}^{1/(1-k)})$ with probability at least $1-delta$.

This page was built for publication: Stochastic Approximation for Risk-aware Markov Decision Processes

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6301452)