Doubly robust policy evaluation and optimization
From MaRDI portal
Abstract: We study sequential decision making in environments where rewards are only partially observed, but can be modeled as a function of observed contexts and the chosen action by the decision maker. This setting, known as contextual bandits, encompasses a wide variety of applications such as health care, content recommendation and Internet advertising. A central task is evaluation of a new policy given historic data consisting of contexts, actions and received rewards. The key challenge is that the past data typically does not faithfully represent proportions of actions taken by a new policy. Previous approaches rely either on models of rewards or models of the past policy. The former are plagued by a large bias whereas the latter have a large variance. In this work, we leverage the strengths and overcome the weaknesses of the two approaches by applying the doubly robust estimation technique to the problems of policy evaluation and optimization. We prove that this approach yields accurate value estimates when we have either a good (but not necessarily consistent) model of rewards or a good (but not necessarily consistent) model of past policy. Extensive empirical comparison demonstrates that the doubly robust estimation uniformly improves over existing techniques, achieving both lower variance in value estimation and better policies. As such, we expect the doubly robust approach to become common practice in policy evaluation and optimization.
Recommendations
Cites work
- A Generalization of Sampling Without Replacement From a Finite Universe
- A new approach to causal inference in mortality studies with a sustained exposure period—application to control of the healthy worker survivor effect
- A robust method for estimating optimal treatment regimes
- Counterfactual reasoning and learning systems: the example of computational advertising
- Demystifying double robustness: a comparison of alternative strategies for estimating a population mean from incomplete data
- Doubly robust policy evaluation and optimization
- Estimation of Regression Coefficients When Some Regressors Are Not Always Observed
- scientific article; zbMATH DE number 6253908 (Why is no real title available?)
- Improving predictive inference under covariate shift by weighting the log-likelihood function
- Marginal Mean Models for Dynamic Regimes
- On model selection and model misspecification in causal inference
- On tail probabilities for martingales
- Optimal Dynamic Treatment Regimes
- Pattern-recognizing stochastic learning automata
- Semiparametric Efficiency in Multivariate Regression Models with Missing Data
- Semiparametric regression estimation in the presence of dependent censoring
- Some aspects of the sequential design of experiments
- Some results on generalized difference estimation and generalized regression estimation for finite populations
- The Nonstochastic Multiarmed Bandit Problem
Cited in
(26)- Constrained Bayesian optimization with noisy experiments
- Importance sampling in reinforcement learning with an estimated behavior policy
- Partially observable environment estimation with uplift inference for reinforcement learning based recommendation
- Optimal policy trees
- Toward theoretical understandings of robust Markov decision processes: sample complexity and asymptotics
- Batch policy learning in average reward Markov decision processes
- PAC-Bayesian lifelong learning for multi-armed bandits
- Augmented direct learning for conditional average treatment effect estimation with double robustness
- Constructing effective personalized policies using counterfactual inference from biased data sets with many features
- Doubly robust policy evaluation and optimization
- More efficient policy learning via optimal retargeting
- Doubly Robust Crowdsourcing
- Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learning
- A Single-Index Model With a Surface-Link for Optimizing Individualized Dose Rules
- Double reinforcement learning for efficient off-policy evaluation in Markov decision processes
- A contextual bandit bake-off
- Bayesian optimization for policy search via online-offline experimentation
- Nonparametric causal effects based on incremental propensity score interventions
- Learning when-to-treat policies
- Statistical inference for online decision making: in a contextual bandit setting
- Selecting and ranking individualized treatment rules with unmeasured confounding
- A multiagent reinforcement learning framework for off-policy evaluation in two-sided markets
- Offline Multi-Action Policy Learning: Generalization and Optimization
- Debiasing in-sample policy performance for small-data, large-scale optimization
- Fighting sampling bias: a framework for training and evaluating credit scoring models
- A review of off-policy evaluation in reinforcement learning
This page was built for publication: Doubly robust policy evaluation and optimization
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q252797)
