Off-Policy Confidence Interval Estimation with Confounded Markov Decision Process
From MaRDI portal
Publication:6153991
Abstract: This paper is concerned with constructing a confidence interval for a target policy's value offline based on a pre-collected observational data in infinite horizon settings. Most of the existing works assume no unmeasured variables exist that confound the observed actions. This assumption, however, is likely to be violated in real applications such as healthcare and technological industries. In this paper, we show that with some auxiliary variables that mediate the effect of actions on the system dynamics, the target policy's value is identifiable in a confounded Markov decision process. Based on this result, we develop an efficient off-policy value estimator that is robust to potential model misspecification and provide rigorous uncertainty quantification. Our method is justified by theoretical results, simulated and real datasets obtained from ridesharing companies. A Python implementation of the proposed procedure is available at https://github.com/Mamba413/cope.
Cites work
- A robust method for estimating optimal treatment regimes
- Asymptotic Statistics
- Causality. Models, reasoning, and inference
- Double/debiased machine learning for treatment and structural parameters
- Estimating dynamic treatment regimes in mobile health using V-learning
- Estimation and Inference of Heterogeneous Treatment Effects using Random Forests
- Evaluating marker-guided treatment selection strategies
- Gaussian approximation of suprema of empirical processes
- scientific article; zbMATH DE number 490141 (Why is no real title available?)
- Multiply Robust Causal Inference with Double-Negative Control Adjustment for Categorical Unmeasured Confounding
- Nonparametric regression using deep neural networks with ReLU activation function
- Off-policy estimation of long-term average outcomes with applications to mobile health
- Optimal Dynamic Treatment Regimes
- Optimal Structural Nested Models for Optimal Sequential Decisions
- Quantile-optimal treatment regimes
- Reinforcement learning. An introduction
- Resampling‐based confidence intervals for model‐free robust inference on optimal treatment regimes
- Robust estimation of optimal dynamic treatment regimes for sequential treatment decisions
- Robust inference on population indirect causal effects: the generalized front door criterion
- Statistical inference for the mean outcome under a possibly non-unique optimal treatment strategy
Cited in
(7)- Reinforcement Learning in Latent Heterogeneous Environments
- Generalized fitted Q-iteration with clustered data
- A semiparametric instrumented difference-in-differences approach to policy learning
- Online statistical inference in decision-making with matrix context
- Reinforcement Learning with Continuous Actions Under Unmeasured Confounding
- A review of off-policy evaluation in reinforcement learning
- Low-Rank Contextual Reinforcement Learning from Heterogeneous Human Feedback
This page was built for publication: Off-Policy Confidence Interval Estimation with Confounded Markov Decision Process
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6153991)
