On the stability of some controlled Markov chains and its applications to stochastic approximation with Markovian dynamic (Q2258523)

A model of controlled Markov chains is considered in which the transition probabilities \(P_\vartheta(x,A)\) depend on some parameter \(\vartheta\). The conditional distribution of the chain at the (\(i+1\))-th step given the past is \(X_{i+1}|(\vartheta_0,X_0,\dots,X_i)\sim P_{\vartheta_i}(X_i,\cdot)\), and \(\vartheta_{i+1}=\varphi_{i+1}(\vartheta_0,X_0,\dots,X_{i+1})\), where \(\varphi_i\), \(i=1,2\dots\) is a family of nonrandom mappings. A Lyapunov function technique is used to establish recurrence of the joint process \((\vartheta_i,X_i)\) to a set. The authors describe how to combine a joint Lyaunov function from two individual functions for \(X\) and \(\vartheta\). The technique applies even in situations where the dynamics exhibits a time-scale separation. The results are applied to the stability analysis of Robbins-Monro stochastic approximation algorithms and adaptive Markov chain Monte Carlo algorithms.