A time aggregation approach to Markov decision processes (Q1614322)

An infinite horizon average cost control problem for discrete time ergodic Markov chains is considered. A time aggregation approach is proposed, by which policy iteration of the original problem is replaced by a series of policy iterations on nonintersecting subsets of the state space, using the associated embedded Markov chains and equivalent performance functions. Single sample path-based estimation algorithms are presented. The results are illustrated by numerical and simulation examples.