Off-policy evaluation in partially observed Markov decision processes under sequential ignorability

DOI10.1214/23-AOS2287MaRDI QIDQ6183750zbMATH OpenFDO

Authors Yu-Chen Hu, Stefan Wager

Publication date 4 January 2024

Published in The Annals of Statistics (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2110.12343, https://projecteuclid.org/journals/annals-of-statistics/volume-51/issue-4/Off-policy-evaluation-in-partially-observed-Markov-decision-processes-under/10.1214/23-AOS2287.full

zbMATH Keywords

causal inferencemixing timeLepski's methodimportance weightingsequential ignorability

Mathematics Subject Classification ID

Non-Markovian processes: estimation (62M09) Causal inference from observational studies (62D20)

Abstract: We consider off-policy evaluation of dynamic treatment rules under sequential ignorability, given an assumption that the underlying system can be modeled as a partially observed Markov decision process (POMDP). We propose an estimator, partial history importance weighting, and show that it can consistently estimate the stationary mean rewards of a target policy given long enough draws from the behavior policy. We provide an upper bound on its error that decays polynomially in the number of observations (i.e., the number of trajectories times their length), with an exponent that depends on the overlap of the target and behavior policies, and on the mixing time of the underlying system. Furthermore, we show that this rate of convergence is minimax given only our assumptions on mixing and overlap. Our results establish that off-policy evaluation in POMDPs is strictly harder than off-policy evaluation in (fully observed) Markov decision processes, but strictly easier than model-free off-policy evaluation.

Recommendations

• Partially observable Markov decision processes with imprecise parameters
• scientific article; zbMATH DE number 1361472
• An incremental off-policy search in a model-free Markov decision process using a single sample path
• Continuous-time Markov decisions based on partial exploration
• Partially observable Markov decision processes with partially observable random discount factors
• scientific article; zbMATH DE number 1509479
• Partially observable total-cost Markov decision processes with weakly continuous transition probabilities
• Partially observed Markov decision processes with binomial observations
• Partially observed Markov decision process multiarmed bandits-structural results
• scientific article; zbMATH DE number 409476

Cites work

• scientific article; zbMATH DE number 7225149 (Why is no real title available?)
• scientific article; zbMATH DE number 783783 (Why is no real title available?)
• scientific article; zbMATH DE number 7306868 (Why is no real title available?)
• scientific article; zbMATH DE number 6542809 (Why is no real title available?)
• A CENTRAL LIMIT THEOREM AND A STRONG MIXING CONDITION
• A generalization error for Q-learning
• A new approach to causal inference in mortality studies with a sustained exposure period—application to control of the healthy worker survivor effect
• Ambiguous partially observable Markov decision processes: structural results and applications
• An alternative point of view on Lepski's method
• Asymptotic Statistics
• Batch policy learning in average reward Markov decision processes
• Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learning
• Estimating dynamic treatment regimes in mobile health using V-learning
• Estimating individualized treatment rules using outcome weighted learning
• From local kernel to nonlocal multiple-model image denoising
• Identifying causal effects with proxy variables of an unmeasured confounder
• Learning near-optimal policies with Bellman-residual minimization based fitted policy iteration and a single sample path
• Learning when-to-treat policies
• Marginal Structural Models to Estimate the Joint Causal Effect of Nonrandomized Treatments
• Off-policy estimation of long-term average outcomes with applications to mobile health
• On spatially adaptive estimation of nonparametric regression
• Optimal Dynamic Treatment Regimes
• Optimal Structural Nested Models for Optimal Sequential Decisions
• Performance guarantees for individualized treatment rules
• Planning and acting in partially observable stochastic domains
• Policy learning with observational data
• Probability Inequalities for Sums of Bounded Random Variables
• Robust estimation of optimal dynamic treatment regimes for sequential treatment decisions
• Spatially Adaptive Estimation via Fitted Local Likelihood Techniques
• State of the Art—A Survey of Partially Observable Markov Decision Processes: Theory, Models, and Algorithms
• The Optimal Control of Partially Observable Markov Processes over a Finite Horizon
• The Optimal Control of Partially Observable Markov Processes over the Infinite Horizon: Discounted Costs
• Unified methods for censored longitudinal data and causality
• Universal Reinforcement Learning
• Weighted sums of certain dependent random variables
• Who should be treated? Empirical welfare maximization methods for treatment choice

Cited in

(3)

• Reliable off-policy evaluation for reinforcement learning
• Proximal reinforcement learning: efficient off-policy evaluation in partially observed Markov decision processes
• Partially observable environment estimation with uplift inference for reinforcement learning based recommendation

This page was built for publication: Off-policy evaluation in partially observed Markov decision processes under sequential ignorability

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6183750)