Necessary conditions for the optimality equation in average-reward Markov decision processes (Q1115358)

The paper considers average reward Markov decision processes with denumerable state space, compact action space and bounded rewards. One condition for the optimality equation to have a bounded solution with constant gain rate is that the return times to a state x(f) are uniformly bounded over all stationary policies f (here condition A). This condition cannot be expected to be necessary since the special reward function is not involved. But it is shown to be necessary for the above property to hold uniformly over all bounded reward functions and all subproblems with restricted action sets (with some relation between the bounds for the rewards and for the solution). A similar question for finite state spaces was discussed by \textit{P. J. Schweitzer} [R.A.I.R.O. 19, 71-86 (1985; Zbl 0571.90095)].