Continuous-time controlled Markov chains. (Q1872339)

The authors consider continuous-time Markov decision processes with a denumerable state space and Borel action space where the cost and the transition rates are allowed to be unbounded. In the class of deterministic stationary policies they first give very weak conditions under which the existence of \(\varepsilon\)-optimal \((\varepsilon\geq 0)\) policies is proved using the construction of a minimum \(Q\)-process. In the class of randomized Markov policies they secondly present a new necessary and sufficient condition for a nonhomogeneous \(Q\)-process to be regular and under this condition the existence of \(\varepsilon\)-optimal stationary policies is proved. In the case of nonregular \(Q\)-process, they further use Kolmogorov's forward equation for a nonhomogeneous minimum \(Q\)-process to prove the existence of \(\varepsilon\)-optimal stationary policies.