Generic E-Variables for Exact Sequential k-Sample Tests that allow for Optional Stopping

DOI10.48550/ARXIV.2106.02693arXiv2106.02693OpenAlexW4388195981MaRDI QIDQ87830FDOQ87830

Authors: Rosanne Turner, Alexander Ly, Peter Grünwald, Rosanne J. Turner, Alexander Ly, Peter D. Grünwald 

Publication date: 4 June 2021

Published in: Journal of Statistical Planning and Inference (Search for Journal in Brave)

Abstract: We develop E-variables for testing whether two or more data streams come from the same source or not, and more generally, whether the difference between the sources is larger than some minimal effect size. These E-variables lead to exact, nonasymptotic tests that remain safe, i.e. keep their type-I error guarantees, under flexible sampling scenarios such as optional stopping and continuation. In special cases our E-variables also have an optimal 'growth' property under the alternative. While the construction is generic, we illustrate it through the special case of k x 2 contingency tables, where we also allow for the incorporation of different restrictions on a composite alternative. Comparison to p-value analysis in simulations and a real-world example show that E-variables, through their flexibility, often allow for early stopping of data collection, thereby retaining similar power as classical methods, while also retaining the option of extending or combining data afterwards.

Full work available at URL: https://arxiv.org/abs/2106.02693

Recommendations

• Exact anytime-valid confidence intervals for contingency tables and beyond
• Sequential tests controlling generalized familywise error rates
• Sequential multiple testing with generalized error control: an asymptotic optimality theory
• Multistage tests of multiple hypotheses
• Optimal stopping rules for sequential hypothesis testing

zbMATH Keywords

hypothesis testing\(E\)-variablescomposite hypothesissequential testtest martingaletype-I error control

Mathematics Subject Classification ID

Statistics (62-XX)

Cites Work

• Sequential analysis. Tests and confidence intervals
• On confidence sequences
• Time-uniform, nonparametric, nonasymptotic confidence sequences
• Log-optimal anytime-valid E-values
• Present Position and Potential Developments: Some Personal Views: Statistical Theory: The Prequential Approach
• Bayes factors for independence in contingency tables
• Title not available (Why is that?)
• Statistical Evidence
• Title not available (Why is that?)
• Almost the best of three worlds: risk, consistency and optional stopping for the switch criterion in nested model selection
• Efficient tracking of a growing number of experts
• Always Valid Inference: Continuous Monitoring of A/B Tests
• Valid sequential inference on probability forecast performance
• A Sequential Non-Parametric Multivariate Two-Sample Test
• CONFIDENCE SEQUENCES FOR MEAN, VARIANCE, AND MEDIAN
• Statistical Methods Related to the Law of the Iterated Logarithm
• Title not available (Why is that?)
• Likelihood, Replicability and Robbins' Confidence Sequences
• Exact anytime-valid confidence intervals for contingency tables and beyond
• Test martingales, Bayes factors and \(p\)-values
• E-values: calibration, combination and applications

Cited In (4)

• E-statistics, group invariance and anytime-valid testing
• safestats
• Game-theoretic statistical inference: optional sampling, universal inference, and multiple testing based on e-values. Abstracts from the workshop held May 5--10, 2024
• Exact anytime-valid confidence intervals for contingency tables and beyond

This page was built for publication: Generic E-Variables for Exact Sequential k-Sample Tests that allow for Optional Stopping

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q87830)