Minimum information divergence of Q-functions for dynamic treatment resumes

DOI10.1007/S41884-022-00084-8MaRDI QIDQ6138797zbMATH OpenOpenAlexFDO

Authors Shinto Eguchi

Publication date 16 January 2024

Published in Information Geometry (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2211.08741

zbMATH Keywords

geometric meaninformation geometryminimum divergenceoptimal policysemiparametric modelvalue function

Mathematics Subject Classification ID

Statistics (62-XX) Sufficiency and information (62Bxx) Parametric inference (62Fxx)

Abstract: This paper aims at presenting a new application of information geometry to reinforcement learning focusing on dynamic treatment resumes. In a standard framework of reinforcement learning, a Q-function is defined as the conditional expectation of a reward given a state and an action for a single-stage situation. We introduce an equivalence relation, called the policy equivalence, in the space of all the Q-functions. A class of information divergence is defined in the Q-function space for every stage. The main objective is to propose an estimator of the optimal policy function by a method of minimum information divergence based on a dataset of trajectories. In particular, we discuss the $gamma$-power divergence that is shown to have an advantageous property such that the $gamma$-power divergence between policy-equivalent Q-functions vanishes. This property essentially works to seek the optimal policy, which is discussed in a framework of a semiparametric model for the Q-function. The specific choices of power index $gamma$ give interesting relationships of the value function, and the geometric and harmonic means of the Q-function. A numerical experiment demonstrates the performance of the minimum $gamma$-power divergence method in the context of dynamic treatment regimes.

Cites work

• A paradox concerning nuisance parameters and projected estimating functions
• Differential geometry of curved exponential families. Curvatures and information loss
• Doubly-robust dynamic treatment regimen estimation via weighted least squares
• Entropy and divergence associated with power function and the statistical application
• Estimating time-varying causal excursion effects in mobile health with binary outcomes
• Geometry of minimum contrast
• scientific article; zbMATH DE number 3126094 (Why is no real title available?)
• scientific article; zbMATH DE number 47310 (Why is no real title available?)
• scientific article; zbMATH DE number 1560711 (Why is no real title available?)
• scientific article; zbMATH DE number 3106666 (Why is no real title available?)
• Information geometry
• Information geometry and its applications
• Minimum Divergence Methods in Statistical Machine Learning
• New statistical learning methods for estimating optimal dynamic treatment regimes
• Optimal Structural Nested Models for Optimal Sequential Decisions
• Partial likelihood
• Reinforcement learning. An introduction
• Robust parameter estimation with a small bias against heavy contamination
• Second order efficiency of minimum contrast estimators in a curved exponential family
• Semiparametric theory and missing data.
• Statistical methods for dynamic treatment regimes. Reinforcement learning, causal inference, and personalized medicine

This page was built for publication: Minimum information divergence of Q-functions for dynamic treatment resumes

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6138797)