BET on independence

DOI10.1080/01621459.2018.1537921zbMATH Open1428.62181arXiv1610.05246OpenAlexW3098615150MaRDI QIDQ5208069FDOQ5208069

Authors: Kai Zhang 

Publication date: 15 January 2020

Published in: Journal of the American Statistical Association (Search for Journal in Brave)

Abstract: We study the problem of nonparametric dependence detection. Many existing methods may suffer severe power loss due to non-uniform consistency, which we illustrate with a paradox. To avoid such power loss, we approach the nonparametric test of independence through the new framework of binary expansion statistics (BEStat) and binary expansion testing (BET), which examine dependence through a novel binary expansion filtration approximation of the copula. Through a Hadamard transform, we find that the symmetry statistics in the filtration are complete sufficient statistics for dependence. These statistics are also uncorrelated under the null. By utilizing symmetry statistics, the BET avoids the problem of non-uniform consistency and improves upon a wide class of commonly used methods (a) by achieving the minimax rate in sample size requirement for reliable power and (b) by providing clear interpretations of global relationships upon rejection of independence. The binary expansion approach also connects the symmetry statistics with the current computing system to facilitate efficient bitwise implementation. We illustrate the BET with a study of the distribution of stars in the night sky and with an exploratory data analysis of the TCGA breast cancer data.

Full work available at URL: https://arxiv.org/abs/1610.05246

Recommendations

• Consistent nonparametric tests of independence
• A multivariate nonparametric test of independence
• Nonparametric Independence Tests: Space Partitioning and Kernel Approaches
• Entropy-based independence test
• Power Assessment of a New Test of Independence

zbMATH Keywords

contingency tablemultiple testingnonparametric inferenceHadamard transformbinary expansion

Mathematics Subject Classification ID

Nonparametric estimation (62G05) Nonparametric hypothesis testing (62G10)

Cites Work

• Measuring and testing dependence by correlation of distances
• Equivalence of distance-based and RKHS-based statistics in hypothesis testing
• Introduction to empirical processes and semiparametric inference
• Bootstrap and randomization tests of some nonparametric hypotheses
• Detecting Novel Associations in Large Data Sets
• Equitability, mutual information, and the maximal information coefficient
• A consistent multivariate test of association based on ranks of distances
• Testing Statistical Hypotheses
• A Non-Parametric Test of Independence
• Global testing under sparse alternatives: ANOVA, multiple comparisons and the higher criticism
• Multiprocess parallel antithetic coupling for backward and forward Markov chain Monte Carlo
• On measures of dependence
• The distance correlation \(t\)-test of independence in high dimension
• Brownian distance covariance
• Kernel-Based Tests for Joint Independence
• Consistent distribution-free \(K\)-sample and independence tests for univariate random variables
• Energy statistics: a class of statistics based on distances
• Title not available (Why is that?)
• Distribution-free tests of independence in high dimensions
• Maximally Selected Chi Square Statistics
• Optimal and fast detection of spatial clusters with scan statistics
• A survey of exact inference for contingency tables. With comments and a rejoinder by the author
• The theory of the design of experiments
• Title not available (Why is that?)
• Title not available (Why is that?)
• Application of Walsh Transform to Statistical Analysis
• A theory of the learnable
• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• The analysis of cross-classified categorical data.
• Monotone dependence
• Title not available (Why is that?)
• A Coincidence-Based Test for Uniformity Given Very Sparsely Sampled Discrete Data
• Generalized Measures of Correlation for Asymmetry, Nonlinearity, and Beyond
• Comment: A Fruitful Resolution to Simpson’s Paradox via Multiresolution Inference
• Fisher Exact Scanning for Dependency
• Title not available (Why is that?)

Cited In (21)

• Pairwise nonlinear dependence analysis of genomic data
• Adaptive test of independence based on HSIC measures
• A Multi-resolution Theory for Approximating Infinite-p-Zero-n: Transitional Inference, Individualized Predictions, and a World Without Bias-Variance Tradeoff
• A new set of tools for goodness-of-fit validation
• Fisher Exact Scanning for Dependency
• Nonparametric Prediction Distribution from Resolution-Wise Regression with Heterogeneous Data
• Rank-based indices for testing independence between two high-dimensional vectors
• The Binary Expansion Randomized Ensemble Test
• On Kim-independence
• Comments on “A Gibbs Sampler for a Class of Random Convex Polytopes”
• A survey of some recent developments in measures of association
• A new coefficient of correlation
• Measuring association with Wasserstein distances
• Rearranged dependence measures
• The Hellinger Correlation
• Mind the independence gap
• Extreme value theory for binary expansion testing
• Equitability, interval estimation, and statistical power
• A new framework for distance and kernel-based metrics in high dimensions
• BET
• Distribution-Free Consistent Independence Tests via Center-Outward Ranks and Signs

Uses Software

• energy
• HHG

This page was built for publication: BET on independence

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5208069)