A version of the Euler equation in discounted Markov decision processes (Q1952742)

The paper deals with optimal control problems in discrete time and with an infinite horizon. The problems are considered by means of the Markov decision processes theory. The optimal control problems are proposed to be solved with the dynamic programming technique. The optimal solution is characterized by a functional equation known as the dynamic programming equation. In simple cases, the value iteration procedure is used to approximate the optimal value function. However, this method does not work in more complicated functional forms. The essential tool used in the paper is the Euler equation, this equation is established and solved (in some cases empirically). The authors present an iterative method of finding the solution of the Euler equation in terms of the value iteration function. Under certain conditions, the validity of the Euler equation can be guaranteed. Using the maximizers' convergence of the optimal policy, the optimal control problem is solved. A linear quadratic problem illustrates the theory.