Off-policy evaluation in partially observed Markov decision processes under sequential ignorability
From MaRDI portal
Publication:6183750
DOI10.1214/23-AOS2287arXiv2110.12343MaRDI QIDQ6183750FDOQ6183750
Authors: Yu-Chen Hu, Stefan Wager 
Publication date: 4 January 2024
Published in: The Annals of Statistics (Search for Journal in Brave)
Abstract: We consider off-policy evaluation of dynamic treatment rules under sequential ignorability, given an assumption that the underlying system can be modeled as a partially observed Markov decision process (POMDP). We propose an estimator, partial history importance weighting, and show that it can consistently estimate the stationary mean rewards of a target policy given long enough draws from the behavior policy. We provide an upper bound on its error that decays polynomially in the number of observations (i.e., the number of trajectories times their length), with an exponent that depends on the overlap of the target and behavior policies, and on the mixing time of the underlying system. Furthermore, we show that this rate of convergence is minimax given only our assumptions on mixing and overlap. Our results establish that off-policy evaluation in POMDPs is strictly harder than off-policy evaluation in (fully observed) Markov decision processes, but strictly easier than model-free off-policy evaluation.
Full work available at URL: https://arxiv.org/abs/2110.12343
Recommendations
- Partially observable Markov decision processes with imprecise parameters
- scientific article; zbMATH DE number 1361472
- An incremental off-policy search in a model-free Markov decision process using a single sample path
- Continuous-time Markov decisions based on partial exploration
- Partially observable Markov decision processes with partially observable random discount factors
- scientific article; zbMATH DE number 1509479
- Partially observable total-cost Markov decision processes with weakly continuous transition probabilities
- Partially observed Markov decision processes with binomial observations
- Partially observed Markov decision process multiarmed bandits-structural results
- scientific article; zbMATH DE number 409476
Cites Work
- State of the Art—A Survey of Partially Observable Markov Decision Processes: Theory, Models, and Algorithms
- The Optimal Control of Partially Observable Markov Processes over the Infinite Horizon: Discounted Costs
- The Optimal Control of Partially Observable Markov Processes over a Finite Horizon
- Planning and acting in partially observable stochastic domains
- Unified methods for censored longitudinal data and causality
- Marginal Structural Models to Estimate the Joint Causal Effect of Nonrandomized Treatments
- Asymptotic Statistics
- A new approach to causal inference in mortality studies with a sustained exposure period—application to control of the healthy worker survivor effect
- Probability Inequalities for Sums of Bounded Random Variables
- Who should be treated? Empirical welfare maximization methods for treatment choice
- Estimating individualized treatment rules using outcome weighted learning
- Performance guarantees for individualized treatment rules
- A generalization error for Q-learning
- A CENTRAL LIMIT THEOREM AND A STRONG MIXING CONDITION
- Optimal Dynamic Treatment Regimes
- Optimal Structural Nested Models for Optimal Sequential Decisions
- Robust estimation of optimal dynamic treatment regimes for sequential treatment decisions
- Title not available (Why is that?)
- Weighted sums of certain dependent random variables
- An alternative point of view on Lepski's method
- On spatially adaptive estimation of nonparametric regression
- From local kernel to nonlocal multiple-model image denoising
- Learning near-optimal policies with Bellman-residual minimization based fitted policy iteration and a single sample path
- Spatially Adaptive Estimation via Fitted Local Likelihood Techniques
- Ambiguous partially observable Markov decision processes: structural results and applications
- Identifying causal effects with proxy variables of an unmeasured confounder
- Policy learning with observational data
- Title not available (Why is that?)
- Estimating dynamic treatment regimes in mobile health using V-learning
- Title not available (Why is that?)
- Batch policy learning in average reward Markov decision processes
- Title not available (Why is that?)
- Off-policy estimation of long-term average outcomes with applications to mobile health
- Learning when-to-treat policies
- Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learning
- Universal Reinforcement Learning
Cited In (2)
This page was built for publication: Off-policy evaluation in partially observed Markov decision processes under sequential ignorability
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6183750)
