Fisher Exact Scanning for Dependency

DOI10.1080/01621459.2017.1397522zbMATH Open1478.62132arXiv1608.07885OpenAlexW2963121232MaRDI QIDQ5229908FDOQ5229908

Author name not available (Why is that?)

Publication date: 19 August 2019

Published in: Journal of the American Statistical Association (Search for Journal in Brave)

Abstract: We introduce a method---called Fisher exact scanning (FES)---for testing and identifying variable dependency that generalizes Fisher's exact test on $2imes2$ contingency tables to $RimesC$ contingency tables and continuous sample spaces. FES proceeds through scanning over the sample space using windows in the form of $2imes2$ tables of various sizes, and on each window completing a Fisher's exact test. Based on a factorization of Fisher's multivariate hypergeometric (MHG) likelihood into the product of the univariate hypergeometric likelihoods, we show that there exists a coarse-to-fine, sequential generative representation for the MHG model in the form of a Bayesian network, which in turn implies the mutual independence (up to deviation due to discreteness) among the Fisher's exact tests completed under FES. This allows an exact characterization of the joint null distribution of the $p$-values and gives rise to an effective inference recipe through simple multiple testing procedures such as v{S}id'{a}k and Bonferroni corrections, eliminating the need for resampling. In addition, FES can characterize dependency through reporting significant windows after multiple testing control. The computational complexity of FES is approximately linear in the sample size, which along with the avoidance of resampling makes it ideal for analyzing massive data sets. We use extensive numerical studies to illustrate the work of FES and compare it to several state-of-the-art methods for testing dependency in both statistical and computational performance. Finally, we apply FES to analyzing a microbiome data set and further investigate its relationship with other popular dependency metrics in that context.

Full work available at URL: https://arxiv.org/abs/1608.07885

Recommendations

• Multi-scale Fisher’s independence test for multivariate dependence
• New upper bounds for tight and fast approximation of Fisher's exact test in dependency rule mining
• A general framework for multiple testing dependence
• The Generalized Fisher's Combination and Accurate P-Value Calculation under Dependence
• A generalization of Fisher's exact test in p×q contingency tables using more concordant relations

zbMATH Keywords

multiple testingtest of independencemassive datascalable inferencemulti-scale scanning

Mathematics Subject Classification ID

Hypothesis testing in multivariate analysis (62H15) Contingency tables (62H17)

Cites Work

• ALGORITHM 643
• Measuring and testing dependence by correlation of distances
• Detecting Novel Associations in Large Data Sets
• Equitability, mutual information, and the maximal information coefficient
• Title not available (Why is that?)
• BET on Independence
• A consistent multivariate test of association based on ranks of distances
• A Non-Parametric Test of Independence
• Brownian distance covariance
• Consistent distribution-free \(K\)-sample and independence tests for univariate random variables
• Generalized R-squared for detecting dependence
• Multiscale likelihood analysis and complexity penalized estimation.
• Optimal and fast detection of spatial clusters with scan statistics
• A Bayesian nonparametric approach to testing for dependence between random variables
• THE COMBINATION OF PROBABILITIES ARISING FROM DATA IN DISCRETE DISTRIBUTIONS
• Adaptive shrinkage in Pólya tree type models

Cited In (8)

• The Binary Expansion Randomized Ensemble Test
• Nonparametric Scanning Tests of Homogeneity for Hierarchical Models with Continuous Covariates
• Comments on “A Gibbs Sampler for a Class of Random Convex Polytopes”
• BET on Independence
• A new framework for distance and kernel-based metrics in high dimensions
• Bayesian nonparametric test for independence between random vectors
• ODC and ROC curves, comparison curves and stochastic dominance
• Distribution-Free Consistent Independence Tests via Center-Outward Ranks and Signs

Uses Software

• FEXACT

This page was built for publication: Fisher Exact Scanning for Dependency

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5229908)