Online Bootstrap Inference For Policy Evaluation In Reinforcement Learning
From MaRDI portal
Abstract: The recent emergence of reinforcement learning has created a demand for robust statistical inference methods for the parameter estimates computed using these algorithms. Existing methods for statistical inference in online learning are restricted to settings involving independently sampled observations, while existing statistical inference methods in reinforcement learning (RL) are limited to the batch setting. The online bootstrap is a flexible and efficient approach for statistical inference in linear stochastic approximation algorithms, but its efficacy in settings involving Markov noise, such as RL, has yet to be explored. In this paper, we study the use of the online bootstrap method for statistical inference in RL. In particular, we focus on the temporal difference (TD) learning and Gradient TD (GTD) learning algorithms, which are themselves special instances of linear stochastic approximation under Markov noise. The method is shown to be distributionally consistent for statistical inference in policy evaluation, and numerical experiments are included to demonstrate the effectiveness of this algorithm at statistical inference tasks across a range of real RL environments.
Recommendations
- Online reinforcement learning using a probability density estimation
- On-line policy gradient estimation with multi-step sampling
- Bayesian optimization for policy search via online-offline experimentation
- Online bootstrap confidence intervals for the stochastic gradient descent estimator
- An Online Policy Gradient Algorithm for Markov Decision Processes with Continuous States and Actions
- Online Reinforcement Learning of Optimal Threshold Policies for Markov Decision Processes
- A basic formula for online policy gradient algorithms
Cites work
- 10.1162/1532443041827907
- A Stochastic Approximation Method
- Acceleration of Stochastic Approximation by Averaging
- An analysis of temporal-difference learning with function approximation
- An emphatic approach to the problem of off-policy temporal-difference learning
- Challenges of real-world reinforcement learning: definitions, benchmarks and analysis
- Constructing dynamic treatment regimes over indefinite time horizons
- Dynamic Causal Effects Evaluation in A/B Testing with a Reinforcement Learning Framework
- Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learning
- Estimating dynamic treatment regimes in mobile health using V-learning
- Generalized TD learning
- scientific article; zbMATH DE number 7626801 (Why is no real title available?)
- Inference and uncertainty quantification for noisy matrix completion
- Markov Chains
- Markov chains and mixing times. With a chapter on ``Coupling from the past by James G. Propp and David B. Wilson.
- Markov Chains and Stochastic Stability
- Moment consistency of the exchangeably weighted bootstrap for semiparametric M-estimation
- Online bootstrap confidence intervals for the stochastic gradient descent estimator
- Reinforcement learning. An introduction
- Stability of Stochastic Approximation under Verifiable Conditions
- Statistical inference for model parameters in stochastic gradient descent
- Statistical inference for online decision making via stochastic gradient descent
- The bootstrap and Edgeworth expansion
- Trajectory averaging for stochastic approximation MCMC algorithms
Cited in
(8)- Scalable inference in functional linear regression with streaming data
- Generalized fitted Q-iteration with clustered data
- Online estimation and inference for robust policy evaluation in reinforcement learning
- Online statistical inference in decision-making with matrix context
- Testing stationarity and change point detection in reinforcement learning
- Multivariate dynamic mediation analysis under a reinforcement learning framework
- A review of causal decision making
- Risk-Sensitive Deep RL: Variance-Constrained Actor-Critic Provably Finds Globally Optimal Policy
This page was built for publication: Online Bootstrap Inference For Policy Evaluation In Reinforcement Learning
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6185586)
