Long-term average cost control problems for continuous time Markov processes: A survey (Q788690)

This paper is a survey of long term average cost control problems, namely the cost is defined by \[ J(v)=\underline{\lim}_{T\to \infty}\frac{1}{T}E\int^{T}_{0}f(y(s,v),v(s))ds, \] where \(y(\cdot,v)\) is the response for control v. The object is to minimize J(v). The following simple example illustrate the kind of phenomena treated in the paper. Example: Let V be the totality of Lipschitz continuous functions. For \(v\in V\), \(y(t,v)\) is the solution of SDE, \[ dy_ x(t)=v(y_ x(t))dt+\sqrt{2}dw(t),\quad y_ x(0)=x(\in R^ 1). \] The discounted cost is given by \(J_ x^{\alpha}(v)=E\int^{\infty}_{0}e^{-\alpha t}(y_ x(t)^ 2+v^ 2(y_ x(t)))dt\). The optimal cost function \(u_{\alpha}(x)=\inf_{v\in V}J_ x^{\alpha}(v)\) satisfies the Hamilton-Jacobi-Bellman (H-J-B) equation \(u_{\alpha}''(x)-\inf_{v\in R^ 1}(vu'_{\alpha}(x)+v^ 2)+\alpha u_{\alpha}=x^ 2\). Actually \(u_{\alpha}(x)=(\sqrt{\alpha^ 2+4}-\alpha)(\frac{x^ 2}{2}+\frac{1}{\alpha})\). Hence, \(\lim_{\alpha \to 0}\alpha u_{\alpha}(x)=2\) and \(W(x)=\lim_{\alpha \to 0}(u_{\alpha}(x)- u_{\alpha}(0))=x^ 2\) satisfies \[ -W''(x)-\inf_{v\in R^ 1}(vW'(x)+v^ 2)+\lambda =x^ 2. \] Moreover, putting \(V_ 0=\{v\in V;\lim_{T\to \infty}\frac{1}{T}EW(y_ x(T,v))=0\}\), \[ \lambda =\inf_{v\in V_ 0}(\lim_{T\to \infty}\frac{1}{T}E\int^{T}_{0}y^ 2_ x(t)+v^ 2(y_ x(t))dt) \] and an optimal control \(\hat v(x)=-x\) is a minimum selection of \((vW'(x)+v^ 2)\). In many cases, the problem relates to a solution \((\lambda,W)\) of the H-J-B equation \(\inf_{v\in K}(A^ vW+f(x,v))- \lambda =0,\) and using the discounted problem a solution is obtained. The author treats three problems: continuous control, stopping problem and impulse control, and discusses some open problems.