scientific article; zbMATH DE number 2062343

Authors Daniel P. Heyman, Matthew J. Sobel

Publication date 25 March 2004

Introductory exposition (textbooks, tutorial papers, etc.) pertaining to probability theory (60-01) Foundations of stochastic processes (60G05) Applications of Markov chains and discrete-time Markov processes on general state spaces (social mobility, learning theory, industrial processes, etc.) (60J20) Introductory exposition (textbooks, tutorial papers, etc.) pertaining to operations research and mathematical programming (90-01) Stochastic programming (90C15) Dynamic programming (90C39) Stochastic games, stochastic differential games (91A15)

Review text

The book is an unabridged re-publication of the classic book published in 1984 by McGraw Hill (see Zbl 0531.90062). It is the sequel of volume I (see Zbl 1072.90001), but it can be used without necessarily going through the first volume. It makes clear the sequel by numbering by C its contents as the first volume was divided into two parts: A and B. The oeuvre starts with the presentation of an overview of both volumes and some suggestions, on how to proceed on their use, are given. It is divided into 9 chapters. The first one is an introduction. In it are prologued the basic concepts on Stochastic Optimization (SO) are treated. Afterwards a chapter is devoted to discuss the `myopic' look of a modeler that considers a simple inventory model disregarding the stochasticity of the problem. The discussion justifies the introduction of Markov Decision Processes (MDP), the theoretical frame allows modeling the effect of advertisement as such in the distribution of the random demand and of the expansion of industrial capacity. The next chapter presents the theoretical aspects of MDP in the cases of finite and infinite horizons as well as the stationary policies. 29 examples are worked out and they provide a useful insight into the practical meaning of the theoretical concepts. Some of them are worked out numerically. The average-reward criterion is deeply studied and the optimality of certain stationary policies and algorithms is derived. The subsequent chapter addresses the characteristic of the continuous time MDP. Semi MDP is introduced for dealing with the continuous case. Semi MDP for the infinite horizon case is introduced. An average-reward criterion is revisited under these considerations and an algorithm is presented. A large explanation of Markov Decision Chains, Contraction Mapping, Fixed Point Theorems, and Inventory closes the chapter. Computational aspects of the solution of MDP is the theme of the other chapter. Linear Programming (LP) is the kernel of different programs that look for the solution of the inappropriateness of the solutions obtained via the use of the basic classic program. The probabilistic considerations associated with the Expected-Present Value LP are established and its behavior with respect to generalized programs and Average Reward LP without unchained assumptions. The improvement of the computing speed is derived. Other particularities of the optimization problems, such as Multicriteria, Aggregation and Parametric solutions are also discussed. The following chapter considers the inventory problem introducing different cases as continuous production models, the effect of switching random variables and expectations, the derivation of deterministic equivalents of the stochastic problem etc. It is followed by a chapter where the qualitative look of MDP is developed (sufficient conditions for the existence of monotone optimal policies, binary decision problems, resource-management models, and convergence in policy). The book ends with the presentation of a game theory framework. The generalizations of MDP are made clearer within it. It allows fixing concepts and results (solution concepts, characteristics of the equilibrium point of the game) at a more abstract level. Two appendices provide the needed mathematical background in Probability and Mathematical Analysis. Each chapter contains a large set of bibliographical sources. The book may be used as a text in advanced courses on Operations Research and is a good reference source for researchers within the field.

Recommendations

Cited in

(19)

This page was built for publication:

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q4459240)