scientific article; zbMATH DE number 420890

From MaRDI portal
Publication:3135096





About Markov Decision Processes (MDP), many books and an immense number of papers have been written. The present book is a remarkable book of a remarkable author. As the author remarks in his foreword, the book leans on three monographs of R. A. Howard, J. van der Wal and L. C. M. Kallenberg, resp., which also, in this order, determine the content of the first four chapters. Chapter 1 nicely makes the reader acquainted with the \(z\)-transform, a method of generating functions. In Chapter 2, the framework of MDP using sequences of random variables (state, action, reward) is built. There are presented the problem classes ``Finite Horizon, ``Discounted, ``Average and ``Absorbed with all of their particularities. Narrowly connected with it is Chapter 3, ``Algorithms. Here, for each problem class, the known methods of value and policy iteration are presented, their advantages and disadvantages are weighted one against the other, and refined techniques (elimination of suboptimal strategies) are given. In Chapter 4, for all problem classes, a transformation into a linear programming problem follows, shortly, because Kallenberg had it done already more comprehensively. Chapter 5 introduces into semi-MDP whereby, for simplicity, sojourn times are assumed as discretely distributed. The often more realistic case that the current state cannot be observed itself but only by an indicator (partial observation) and also the case when transition law and reward still depend on an unknown parameter (but with known distribution) being estimated during the observation (adaptive MDP) has been considered in Chapter 6. Chapter 7 simply named ``Further Aspects of MDP reflects original ideas of the author published in his numerous earlier papers. This concerns structured policies, approximation modelling, sensitivity analysis, multiobjective MDP, mean-variance criteria as well as Markov games. After that already earlier applications are named, Chapter 8 finally, at a scarce amount, describes real-life examples which can be formulated as MDP. The book is compact, written naturally and in an ingenious style. It has an entirely proper terminology (Result instead of Proposition) and symbolic (limit instead of lim). It refers to results of earlier research (especially also to the author's book ``Finite Dynamic Programming (1978; Zbl 0429.90043) without extending the list of references (79 items) too much. It is a compendium for students of many faculties as mathematics, statistics, operations research, systems and industrial engineering as well as management science but also lectures and researchers in those field would profit from its use. It is also suited for engineers, because the author often skips mathematical difficulties (measurability, Markov property). For all but one chapter, there exist exercises with solutions, carefully elaborated. Only a few critical remarks may be given: In formulae with ratios, the fraction bars are put in a rather discordant way \((/,-)\). The notation of a conditional random variable in formula (2.24) seems doubtful. Also the book is not entirely free of misprints.




Cited in
(34)








This page was built for publication:

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q3135096)