Projected state-action balancing weights for offline reinforcement learning
From MaRDI portal
(Redirected from Publication:6183753)
Abstract: Offline policy evaluation (OPE) is considered a fundamental and challenging problem in reinforcement learning (RL). This paper focuses on the value estimation of a target policy based on pre-collected data generated from a possibly different policy, under the framework of infinite-horizon Markov decision processes. Motivated by the recently developed marginal importance sampling method in RL and the covariate balancing idea in causal inference, we propose a novel estimator with approximately projected state-action balancing weights for the policy value estimation. We obtain the convergence rate of these weights and show that the proposed value estimator is semi-parametric efficient under technical conditions. In terms of asymptotics, our results scale with both the number of trajectories and the number of decision points at each trajectory. As such, consistency can still be achieved with a limited number of subjects when the number of decision points diverges. In addition, we develop a necessary and sufficient condition for establishing the well-posedness of the Bellman operator in the off-policy setting, which characterizes the difficulty of OPE and may be of independent interest. Numerical experiments demonstrate the promising performance of our proposed estimator.
Recommendations
- Double reinforcement learning for efficient off-policy evaluation in Markov decision processes
- Proximal reinforcement learning: efficient off-policy evaluation in partially observed Markov decision processes
- Reliable off-policy evaluation for reinforcement learning
- Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learning
- Gradient temporal-difference learning for off-policy evaluation using emphatic weightings
Cites work
- A distribution-free theory of nonparametric regression
- Approximate residual balancing: debiased inference of average treatment effects in high dimensions
- Batch policy learning in average reward Markov decision processes
- Covariate balancing propensity score
- Demystifying double robustness: a comparison of alternative strategies for estimating a population mean from incomplete data
- Double reinforcement learning for efficient off-policy evaluation in Markov decision processes
- Dynamic treatment regimes: technical challenges and applications
- Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learning
- Estimating dynamic treatment regimes in mobile health using V-learning
- Estimation of Regression Coefficients When Some Regressors Are Not Always Observed
- Generalized optimal matching methods for causal inference
- Globally efficient non-parametric inference of average treatment effects by empirical balancing calibration weighting
- High-dimensional \(A\)-learning for optimal dynamic treatment regimes
- scientific article; zbMATH DE number 5654889 (Why is no real title available?)
- scientific article; zbMATH DE number 45848 (Why is no real title available?)
- scientific article; zbMATH DE number 3535015 (Why is no real title available?)
- scientific article; zbMATH DE number 700091 (Why is no real title available?)
- Instrumental Variable Estimation of Nonparametric Models
- Kernel-based covariate functional balancing for observational studies
- Large Sample Properties of Generalized Method of Moments Estimators
- Marginal Mean Models for Dynamic Regimes
- Minimal dispersion approximately balancing weights: asymptotic properties and practical considerations
- New statistical learning methods for estimating optimal dynamic treatment regimes
- Nonparametric estimation of an additive model with a link function
- Off-policy estimation of long-term average outcomes with applications to mobile health
- Optimal Dynamic Treatment Regimes
- Optimal global rates of convergence for nonparametric regression
- Optimal sup-norm rates and uniform inference on nonlinear functionals of nonparametric IV regression
- Personalized Policy Learning Using Longitudinal Mobile Health Data
- Quantile-optimal treatment regimes
- Regularized least-squares regression: learning from a sequence
- Regularized policy iteration with nonparametric function spaces
- Some new asymptotic theory for least squares series: pointwise and uniform results
Cited in
(9)- Double reinforcement learning for efficient off-policy evaluation in Markov decision processes
- Offline reinforcement learning with representations for actions
- Off-policy evaluation for tabular reinforcement learning with synthetic trajectories
- Exploiting action impact regularity and exogenous state variables for offline reinforcement learning
- Reliable off-policy evaluation for reinforcement learning
- Proximal reinforcement learning: efficient off-policy evaluation in partially observed Markov decision processes
- Generalized fitted Q-iteration with clustered data
- Testing stationarity and change point detection in reinforcement learning
- Multivariate dynamic mediation analysis under a reinforcement learning framework
This page was built for publication: Projected state-action balancing weights for offline reinforcement learning
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6183753)
