Discrete-time Markov chains. Two-time-scale methods and applications. (Q4831226)

The book is divided into three parts and an appendix. Part I (consisting of Chapters 1 and 2) provides motivational examples, an overview of the book and the background material on discrete-time Markov chains, discrete-time martingales and Gaussian, diffusion and switching diffusion processes. Part II (consisting of Chapters 3--6) treats asymptotic properties of Markov chains. Chapter 3 studies the transition probabilities of discrete-time Markov chains and develops asymptotic expansions composed of two parts: an outer expansion and an initial layer correction. Chapters 4 and 5 deal with occupation measures and exponential bounds, respectively. The last chapter of this part provides an interim summary and extensions. Part III (consisting of Chapters 7--13) presents applications of the theory of Part II in different contexts. Chapter 7 deals with dynamic systems modelled as difference equations subject to regime changes that are modulated by discrete-time Markov chains. The focus here is on the investigation of the long-time behaviour of these systems. Chapter 8 treats hybrid filtering in discrete time. The aim is to reduce the complexity of filtering problems involving large-scale hidden Markov chains. Here a limit system is derived and used to construct approximate filtering systems which are nearly optimal. Similar issues are discussed in the following chapters. In Chapter 9 Markov decision processes involving weak and strong interactions are studied. Chapter 10 is concerned with LQ controls and Chapter 11 with mean-variance controls. In Chapter 12 near-optimal production planning is dicussed. The last chapter of this part is concerned with stochastic approximation. The appendix provides background information on sample paths of Markov chains, weak convergence, and optimal control and HB equations. Each chapter concludes with a ``notes'' section, which provides information on historical developments, as well as references. The book clearly should be of interest to researchers in stochastic control, operations research, manufacturing systems, and applied mathematics. Parts of the book could be used in courses on stochastic control and optimization, or stochastic modelling and analysis for graduate students in mathematics, engineering or economics.