Generic E-Variables for Exact Sequential k-Sample Tests that allow for Optional Stopping

DOI10.48550/ARXIV.2106.02693MaRDI QIDQ87830zbMATH OpenOpenAlexFDO

Authors Rosanne Turner, Alexander Ly, Peter Grünwald, Rosanne J. Turner, Alexander Ly, Peter D. Grünwald

Publication date 4 June 2021

Published in Journal of Statistical Planning and Inference (Search for Journal in Brave)

Copyright license Creative Commons Attribution 4.0 International

Full work available at URL https://arxiv.org/abs/2106.02693

zbMATH Keywords

hypothesis testing\(E\)-variablescomposite hypothesissequential testtest martingaletype-I error control

Mathematics Subject Classification ID

Statistics (62-XX)

Abstract: We develop E-variables for testing whether two or more data streams come from the same source or not, and more generally, whether the difference between the sources is larger than some minimal effect size. These E-variables lead to exact, nonasymptotic tests that remain safe, i.e. keep their type-I error guarantees, under flexible sampling scenarios such as optional stopping and continuation. In special cases our E-variables also have an optimal 'growth' property under the alternative. While the construction is generic, we illustrate it through the special case of k x 2 contingency tables, where we also allow for the incorporation of different restrictions on a composite alternative. Comparison to p-value analysis in simulations and a real-world example show that E-variables, through their flexibility, often allow for early stopping of data collection, thereby retaining similar power as classical methods, while also retaining the option of extending or combining data afterwards.

Recommendations

• Exact anytime-valid confidence intervals for contingency tables and beyond
• Sequential tests controlling generalized familywise error rates
• Sequential multiple testing with generalized error control: an asymptotic optimality theory
• Multistage tests of multiple hypotheses
• Optimal stopping rules for sequential hypothesis testing

Cites work

• scientific article; zbMATH DE number 409719 (Why is no real title available?)
• scientific article; zbMATH DE number 480237 (Why is no real title available?)
• scientific article; zbMATH DE number 3062496 (Why is no real title available?)
• A Sequential Non-Parametric Multivariate Two-Sample Test
• Almost the best of three worlds: risk, consistency and optional stopping for the switch criterion in nested model selection
• Always Valid Inference: Continuous Monitoring of A/B Tests
• Bayes factors for independence in contingency tables
• CONFIDENCE SEQUENCES FOR MEAN, VARIANCE, AND MEDIAN
• E-values: calibration, combination and applications
• Efficient tracking of a growing number of experts
• Exact anytime-valid confidence intervals for contingency tables and beyond
• Likelihood, Replicability and Robbins' Confidence Sequences
• Log-optimal anytime-valid E-values
• On confidence sequences
• Present Position and Potential Developments: Some Personal Views: Statistical Theory: The Prequential Approach
• Sequential analysis. Tests and confidence intervals
• Statistical Evidence
• Statistical Methods Related to the Law of the Iterated Logarithm
• Test martingales, Bayes factors and \(p\)-values
• Time-uniform, nonparametric, nonasymptotic confidence sequences
• Valid sequential inference on probability forecast performance

Cited in

(4)

• safestats
• E-statistics, group invariance and anytime-valid testing
• Game-theoretic statistical inference: optional sampling, universal inference, and multiple testing based on e-values. Abstracts from the workshop held May 5--10, 2024
• Exact anytime-valid confidence intervals for contingency tables and beyond

This page was built for publication: Generic E-Variables for Exact Sequential k-Sample Tests that allow for Optional Stopping

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q87830)