On-line policy gradient estimation with multi-step sampling (Q5962027)

The authors discuss the problem of sample-path-based (on-line) performance gradient estimation for Markov systems. The existing on-line performance gradient estimation algorithms generally require a standard importance sampling assumption. Examples are given to illustrate that the existing on-line policy gradient approaches cannot provide an accurate gradient estimate when the assumption does not hold. It is shown that this assumption can be relaxed and a few new algorithms are proposed based on multi-step sampling. These algorithms do not require this assumption. All the algorithms can be implemented on sample paths and policy gradients can be estimated on-line.