BET on independence

DOI10.1080/01621459.2018.1537921zbMATH Open1428.62181OpenAlexW3098615150MaRDI QIDQ5208069FDOQ5208069

Authors Kai Zhang 

Publication date 15 January 2020

Published in Journal of the American Statistical Association (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1610.05246

zbMATH Keywords

contingency tablemultiple testingnonparametric inferenceHadamard transformbinary expansion

Mathematics Subject Classification ID

Nonparametric estimation (62G05) Nonparametric hypothesis testing (62G10)

Abstract: We study the problem of nonparametric dependence detection. Many existing methods may suffer severe power loss due to non-uniform consistency, which we illustrate with a paradox. To avoid such power loss, we approach the nonparametric test of independence through the new framework of binary expansion statistics (BEStat) and binary expansion testing (BET), which examine dependence through a novel binary expansion filtration approximation of the copula. Through a Hadamard transform, we find that the symmetry statistics in the filtration are complete sufficient statistics for dependence. These statistics are also uncorrelated under the null. By utilizing symmetry statistics, the BET avoids the problem of non-uniform consistency and improves upon a wide class of commonly used methods (a) by achieving the minimax rate in sample size requirement for reliable power and (b) by providing clear interpretations of global relationships upon rejection of independence. The binary expansion approach also connects the symmetry statistics with the current computing system to facilitate efficient bitwise implementation. We illustrate the BET with a study of the distribution of stars in the night sky and with an exploratory data analysis of the TCGA breast cancer data.

Recommendations

• Consistent nonparametric tests of independence
• A multivariate nonparametric test of independence
• Nonparametric Independence Tests: Space Partitioning and Kernel Approaches
• Entropy-based independence test
• Power Assessment of a New Test of Independence

Cites work

• scientific article; zbMATH DE number 3117087 (Why is no real title available?)
• scientific article; zbMATH DE number 3143963 (Why is no real title available?)
• scientific article; zbMATH DE number 49667 (Why is no real title available?)
• scientific article; zbMATH DE number 2172949 (Why is no real title available?)
• scientific article; zbMATH DE number 218925 (Why is no real title available?)
• scientific article; zbMATH DE number 947415 (Why is no real title available?)
• scientific article; zbMATH DE number 5023091 (Why is no real title available?)
• scientific article; zbMATH DE number 3292518 (Why is no real title available?)
• A Coincidence-Based Test for Uniformity Given Very Sparsely Sampled Discrete Data
• A Non-Parametric Test of Independence
• A consistent multivariate test of association based on ranks of distances
• A survey of exact inference for contingency tables. With comments and a rejoinder by the author
• A theory of the learnable
• Application of Walsh Transform to Statistical Analysis
• Bootstrap and randomization tests of some nonparametric hypotheses
• Brownian distance covariance
• Comment: A Fruitful Resolution to Simpson’s Paradox via Multiresolution Inference
• Consistent distribution-free \(K\)-sample and independence tests for univariate random variables
• Detecting novel associations in large data sets
• Distribution-free tests of independence in high dimensions
• Energy statistics: a class of statistics based on distances
• Equitability, mutual information, and the maximal information coefficient
• Equivalence of distance-based and RKHS-based statistics in hypothesis testing
• Fisher Exact Scanning for Dependency
• Generalized Measures of Correlation for Asymmetry, Nonlinearity, and Beyond
• Global testing under sparse alternatives: ANOVA, multiple comparisons and the higher criticism
• Introduction to empirical processes and semiparametric inference
• Kernel-based tests for joint independence
• Maximally Selected Chi Square Statistics
• Measuring and testing dependence by correlation of distances
• Monotone dependence
• Multiprocess parallel antithetic coupling for backward and forward Markov chain Monte Carlo
• On measures of dependence
• Optimal and fast detection of spatial clusters with scan statistics
• Testing Statistical Hypotheses
• The analysis of cross-classified categorical data.
• The distance correlation \(t\)-test of independence in high dimension
• The theory of the design of experiments

Cited in

(21)

• The Hellinger Correlation
• A survey of some recent developments in measures of association
• Adaptive test of independence based on HSIC measures
• A new coefficient of correlation
• Mind the independence gap
• On Kim-independence
• Extreme value theory for binary expansion testing
• Fisher Exact Scanning for Dependency
• A multi-resolution theory for approximating infinite-\(p\)-zero-\(n\): transitional inference, individualized predictions, and a world without bias-variance tradeoff
• A new framework for distance and kernel-based metrics in high dimensions
• Equitability, interval estimation, and statistical power
• Pairwise nonlinear dependence analysis of genomic data
• A new set of tools for goodness-of-fit validation
• Distribution-Free Consistent Independence Tests via Center-Outward Ranks and Signs
• Measuring association with Wasserstein distances
• BET
• Nonparametric Prediction Distribution from Resolution-Wise Regression with Heterogeneous Data
• Rearranged dependence measures
• Rank-based indices for testing independence between two high-dimensional vectors
• The Binary Expansion Randomized Ensemble Test
• Comments on “A Gibbs Sampler for a Class of Random Convex Polytopes”

Describes a project that uses

Uses Software

• energy
• HHG

This page was built for publication: BET on independence

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5208069)