Mathematical foundations of reinforcement learning
The book gives a mathematical introduction to concepts, problems and algorithms in reinforcement learning, intended for graduate and senior undergraduate students, as well as for practitioners. After an introductory overview of its content, the book is divided into ten chapters, complemented with four mathematical appendices. The first three chapters introduce the basic concepts underlying reinforcement learning, in particular states, actions, rewards, returns, state values and policies, as well as the basic theoretical result underlying search for optimal policies -- the Bellman optimality equation. The following two chapters introduce the basic model-based dynamic programming algorithms and the simplest model-free algorithms. The former are based on value iteration and policy iteration, demonstrating the iterative interplay between value evaluation and policy improvement. The latter introduce the application of Monte Carlo methods. A specific role has the sixth chapter, which does not deal directly with reinforcement learning, but introduces the stochastic approximation paradigm, needed in the remaining four chapters of the book. The most extensive chapters of the book are Chapters~7 and 8. The former introduces temporal difference methods, both those learning state values and those learning action values, including the probably most popular reinforcement learning method, the Q-learning of optimal action values. The latter deals with methods based on function approximation, both those learning state values and those learning action values. The final chapters move on to modern techniques: the topic of Chapter~9 are policy gradient methods, that of Chapter~10 are actor-critic frameworks.\N\NAlthough the fundamentals of reinforcement learning are explained with mathematical rigor, the most conspicuous feature of the book is their pedagogical presentation. It isolates core ideas of algorithms, focusing the main parts of the text on them, separated from complementary parts containing proofs. It also guides the reader from more common to more specific concepts, for example, from recursive formulations to value functions and optimal policies. The pedagogical aspect also motivated the many illustrative examples presented in a simple grid-world environment. However, it may also be the reason why important equations are repeated most of the times when they are needed, although for readers at the level of graduate and senior undergraduate students, simply referring to a previous occurrence of the respective equation might be sufficient.
This page was built for publication: Mathematical foundations of reinforcement learning
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q7021927)
