Discrete Dynamic Programming with Sensitive Discount Optimality Criteria

From MaRDI portal





A sequence of sensitive discount optimality criteria, unifying two criteria previously suggested by Blackwell, is introduced for both continuous and discrete time parameter finite Markov decision chains. Let \( V_{\rho}(\pi) \) denote the expected discounted reward under the policy \( \pi \) where \( \rho \) is the rate of interest. For each \( n=-1,0, \ldots \), a policy \( \pi^{*} \) is called \( n^{ \pm} \)optimal if \[ \liminf _{\rho \rightarrow 0 \pm}|\rho|^{-n_{\rho}}\left[V_{\rho}\left(\pi^{*}\right)-V_{\rho}(\pi)\right] \geq 0 \] for all \( \pi \), and \(\pm\) optimal if \( V_{\rho}\left(\pi^{*}\right)-V_{\rho}(\pi) \geq 0 \) for all \( \pi \) and all \( 0< \pm \rho<\rho^{*} \) for some \( \rho^{*}>0 \). When \( n \) is as large as the number of states, \( n^{ \pm} \)optimality is shown to be equivalent to \(\pm\) optimality. The recent policy improvement algorithm of Miller and the author for finding \(\pm\) optimal policies is refined and simplified to find \( n^{ \pm} \)optimal policies. The algorithm exploits the fact that for stationary policies the Laurent expansion about the origin of the return \( V_{p}(\cdot))\) has a simple form. Detailed properties of the coefficients of this expansion are exhibited. Resolvents of substochastic matrices play a central role in this analysis. The methods of successive approximation (Shapley) and policy improvement (Howard) are applied to finding a policy \( \pi \) maximizing \( V_{\rho}(\pi) \) for a fixed negative \( p \), provided the series defining \( V_{\rho}(\cdot) \) is absolutely convergent for all stationary policies.





Cited in
(70)








This page was built for publication: Discrete Dynamic Programming with Sensitive Discount Optimality Criteria

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5575236)