Nearly optimal controls of Markovian systems. (Q4407572)

Part 1-1. A singularly perturbed discrete time Markov chain \((\alpha^\varepsilon_n, \, n\geq 0)\) on a finite state space with a general (i.e. not quadratic) cost functional and a transition matrix \(P^\varepsilon(u) = P(u)+\varepsilon Q(u)\) is under consideration on the infinite horizon. Here \(\varepsilon\) is a small parameter, \(u(\cdot)\) defines an admissible control with values in the compact subset of a Euclidean space \(\Gamma\); \(P\) and \(P^\varepsilon\) are probability matrices. The discounted cost functional to be minimized is \(E\left(\varepsilon \sum_{n=0}^\infty (1-\beta\varepsilon)^n g(\alpha^\varepsilon_n, u(\alpha^\varepsilon_n)) \right)\). Although in principle this problem can be solved using standard linear programming technique, in practice this is useless for large dimensions. It is shown that as \(\varepsilon \to 0\), the original control problem approximates another much simpler one. Once this limit problem is solved, it is shown that the approximate cost functions converge to the limiting one. Next, the optimal control is constructed for the limit system, and it is shown that it remains asymptotically optimal for the perturbed chain. Part 1-2. Another `parallel' problem is studied, with a `long-run average cost', \[ \limsup_{N\to\infty} N^{-1}E\left(\sum_{n=0}^N g(\alpha^\varepsilon_n, u(\alpha^\varepsilon_n)) \right). \] Again, it is shown that a limiting control problem exists, and its optimal control is asymptotically optimal for the perturbed chain. Slightly different assumptions are imposed here. Generalized models are considered under relaxed assumptions for both settings; proofs are omitted with a reference to the authors' paper [Appl. Math. Optimization 44, No. 2, 105--129 (2001; Zbl 0990.90125)]. Part 2. In this part `hybrid' discrete time linear-quadratic (LQ) problems are considered. This means a `simple' LQ problem with frequent switching governed by an `independent' Markov chain. The computational difficulties while solving such a problem is due again to large dimensions. Hence, the idea is to use aggregation which might reduce dimension. As in the part one, a time step \(\varepsilon\) of the system is small, and the switching Markov chain depends on this parameter, too. It is shown that there exists a limit continuous time system which is easier for the analysis. The asymptotic optimality of the limit optimal control for the perturbed chain is established. More general systems are considered with statements and without proofs (a reference on a preprint which is likely to have been printed as [Automatica 38, No. 3, 409--419 (2002; Zbl 1081.93528)] is given). A numerical example is provided. For the entire collection see [Zbl 1005.00025].