Discrete-time Markov chains. Two-time-scale methods and applications.
Research exposition (monographs, survey articles) pertaining to probability theory (60-02) Markov chains (discrete-time Markov processes on discrete state spaces) (60J10) Applications of Markov chains and discrete-time Markov processes on general state spaces (social mobility, learning theory, industrial processes, etc.) (60J20) Research exposition (monographs, survey articles) pertaining to operations research and mathematical programming (90-02) Production models (90B30) Dynamic programming (90C39) Markov and semi-Markov decision processes (90C40) Optimal stochastic control (93E20)
The book is divided into three parts and an appendix. Part I (consisting of Chapters 1 and 2) provides motivational examples, an overview of the book and the background material on discrete-time Markov chains, discrete-time martingales and Gaussian, diffusion and switching diffusion processes. Part II (consisting of Chapters 3--6) treats asymptotic properties of Markov chains. Chapter 3 studies the transition probabilities of discrete-time Markov chains and develops asymptotic expansions composed of two parts: an outer expansion and an initial layer correction. Chapters 4 and 5 deal with occupation measures and exponential bounds, respectively. The last chapter of this part provides an interim summary and extensions. Part III (consisting of Chapters 7--13) presents applications of the theory of Part II in different contexts. Chapter 7 deals with dynamic systems modelled as difference equations subject to regime changes that are modulated by discrete-time Markov chains. The focus here is on the investigation of the long-time behaviour of these systems. Chapter 8 treats hybrid filtering in discrete time. The aim is to reduce the complexity of filtering problems involving large-scale hidden Markov chains. Here a limit system is derived and used to construct approximate filtering systems which are nearly optimal. Similar issues are discussed in the following chapters. In Chapter 9 Markov decision processes involving weak and strong interactions are studied. Chapter 10 is concerned with LQ controls and Chapter 11 with mean-variance controls. In Chapter 12 near-optimal production planning is dicussed. The last chapter of this part is concerned with stochastic approximation. The appendix provides background information on sample paths of Markov chains, weak convergence, and optimal control and HB equations. Each chapter concludes with a ``notes section, which provides information on historical developments, as well as references. The book clearly should be of interest to researchers in stochastic control, operations research, manufacturing systems, and applied mathematics. Parts of the book could be used in courses on stochastic control and optimization, or stochastic modelling and analysis for graduate students in mathematics, engineering or economics.
- Continuous-time Markov chains and applications. A two-time-scale approach
- Markov Processes and Applications
- Markov chains and stochastic stability
- Continuous-time Markov decision processes. Borel space models and general control strategies. With a foreword by Albert Nikolaevich Shiryaev
- Markov chains. Theory and applications
- How does a stochastic optimization/approximation algorithm adapt to a randomly evolving optimum/root with jump Markov sample paths
- Tracking and identification of regime-switching systems using binary sensors
- Bounding the lumping error in Markov chain dynamics
- Multiple time scale decomposition of discrete time Markov chains
- Dissipative control of a three-species food chain stochastic system with a hidden Markov chain
- Fast computation of stationary joint probability distribution of sparse Markov chains
- Stochastic systems arising from Markov modulated empirical measures
- Integration by parts and martingale representation for a Markov chain
- Ergodic theory for controlled Markov chains with stationary inputs
- Analytic approximations of queues with lightly- and heavily-correlated autoregressive service times
- Discrete-time hybrid control in Borel spaces
- Perturbed Markov chains with damping component
- Asymptotic properties of consensus-type algorithms for networked systems with regime-switching topologies
- Asymptotic expansions for stationary distributions of nonlinearly perturbed semi-Markov processes. I.
- From random Poincaré maps to stochastic mixed-mode-oscillation patterns
- Convergence of Markov chain approximation on generalized HJB equation and its applications
- Adaptive stepsize selection for tracking in a regime-switching environment
- On the notion of weak stability and related issues of hybrid diffusion systems
- Persistent tracking and identification of regime-switching systems with structural uncertainties: unmodeled dynamics, observation bias, and nonlinear model mismatch
- Optimal stopping of Markov switching Lévy processes
- Discrete-time semi-Markov random evolutions -- average and diffusion approximation of difference equations and additive functionals
- A stochastic approximation algorithm for option pricing model calibration with a switchable market
- Asymptotic properties of Markov-modulated random sequences with fast and slow timescales
- Weak convergence of Markov-modulated random sequences
- Asymptotic expansions for power-exponential moments of hitting times for nonlinearly perturbed semi-Markov processes
- Coupling and ergodic theorems for Markov chains with damping component
- Discrete-time Markov chains with two-time scales and a countable state space: limit results and queueing applications
- Stability of Discrete-Time Regime-Switching Dynamic Systems with Delays
- Stochastic competitive Lotka-Volterra ecosystems under partial observation: feedback controls for permanence and extinction
- System identification: regime switching, unmodeled dynamics, and binary sensors
- Asymptotic properties of hybrid random processes modulated by Markov chains
- Continuous-time Markov chains and applications. A two-time-scale approach
- Pathwise convergence rates for numerical solutions of Markovian switching stochastic differential equations
- Discrete-time semi-Markov random evolutions and their applications
- Foundations of average-cost nonhomogeneous controlled Markov chains
- A fluid-diffusion-hybrid limiting approximation for priority systems with fast and slow customers
- Singular perturbation for the discounted continuous control of piecewise deterministic Markov processes
- Asymptotic optimality for consensus-type stochastic approximation algorithms using iterate averaging
- Distribution of PageRank Mass Among Principle Components of the Web
- Numerical solutions for jump-diffusions with regime switching
- Safety verification for regime-switching jump diffusions via barrier certificates
- Sequences of random matrices modulated by a discrete-time Markov chain*
- Flows of rare events for regularly perturbed semi-Markov processes
- A novel HMM-based learning framework for improving dynamic wireless push system performance
- Discrete-time linear skew-Gaussian system and its recursive fixed-dimensional exact density filtering
- Markov chains and stochastic stability
- Reducing metastable continuous-space Markov chains to Markov chains on a finite set
- A kernel-based stochastic approximation framework for contextual optimization
- Strong convergence of truncated implicit Euler-Maruyama algorithms for the Wonham filters
- Numerical methods for controlled regime-switching diffusions and regime-switching jump diffusions
- Option pricing in a regime-switching model using the fast Fourier transform
- Stochastic optimization algorithms for barrier dividend strategies
- Balanced realizations of regime-switching linear systems
This page was built for publication: Discrete-time Markov chains. Two-time-scale methods and applications.
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q4831226)
