Remarks on the existence of solutions to the average cost optimality equation in Markov decision processes (Q1814435)

Considering Markov decision processes with Borel state and action spaces, the paper deals with the `` average cost optimality equation'' and presents necessary conditions for the existence of a bounded solution to this equation. Roughly spoken, the long-run expected average cost incurred by policy \(\mu\) is given by \[ J(x,\mu):=\limsup_{n\to\infty}{1\over n+1} E^ \mu_ x\left[ \sum_{t=0}^ n c(X_ t,U_ t)\right] \] for state \(X\), control \(U\), cost function \(c\) and initial state \(x\). The interest in finding conditions that guarantee solutions to the average cost optimality equation derives from a result showing that a bounded solution of that equation leads to, e.g., optimal stationary policies for the decision process. The objective of the paper is to exhibit some necessary conditions that complement known sufficient conditions. The authors stress the fact that from their results it can be appreciated more clearly how restrictive it is to require bounded solutions to the average cost optimality equation, which in turn motivates further studies dealing with unbounded solutions.