Comparing distributions by multiple testing across quantiles or CDF values

DOI10.1016/J.JECONOM.2018.04.003MaRDI QIDQ1668577zbMATH OpenOpenAlexWikidataFDO

Authors M. Goldman, David M. Kaplan

Publication date 29 August 2018

Published in Journal of Econometrics (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1708.04658

zbMATH Keywords

Dirichletfamilywise error rateKolmogorov-Smirnov testprobability integral transformregression discontinuity

Mathematics Subject Classification ID

Nonparametric hypothesis testing (62G10) Paired and multiple comparisons; multiple testing (62J15)

Abstract: When comparing two distributions, it is often helpful to learn at which quantiles or values there is a statistically significant difference. This provides more information than the binary "reject" or "do not reject" decision of a global goodness-of-fit test. Framing our question as multiple testing across the continuum of quantiles $auin(0,1)$ or values $rinmathbbR$, we show that the Kolmogorov--Smirnov test (interpreted as a multiple testing procedure) achieves strong control of the familywise error rate. However, its well-known flaw of low sensitivity in the tails remains. We provide an alternative method that retains such strong control of familywise error rate while also having even sensitivity, i.e., equal pointwise type I error rates at each of $noinfty$ order statistics across the distribution. Our one-sample method computes instantly, using our new formula that also instantly computes goodness-of-fit $p$-values and uniform confidence bands. To improve power, we also propose stepdown and pre-test procedures that maintain control of the asymptotic familywise error rate. One-sample and two-sample cases are considered, as well as extensions to regression discontinuity designs and conditional distributions. Simulations, empirical examples, and code are provided.

Recommendations

• Comparison of multivariate distributions using quantile-quantile plots and related tests
• Testing the equality of quantiles for several normal populations
• Multisample Tests for the Equality of Dispersions Based on Quantiles
• Tests for successive differences of quantiles
• Dispersive comparison of distributions: A multisample testing problem
• Tests for dispersive comparison of two distributions

Cites work

• A Bayesian method for weighted sampling
• An improved bootstrap test of stochastic dominance
• Approximate permutation tests and induced order statistics in the regression discontinuity design
• Asymptotic Statistics
• Asymptotic Theory of Certain "Goodness of Fit" Criteria Based on Stochastic Processes
• Fast calculation of boundary crossing probabilities for Poisson processes
• Fractional order statistic approximation for nonparametric conditional quantile inference
• Generalizations of the familywise error rate
• Goodness-of-fit test statistics that dominate the Kolmogorov statistics
• Heterogeneous impacts in PROGRESA
• Histospline smoothing the Bayesian bootstrap
• scientific article; zbMATH DE number 3624650 (Why is no real title available?)
• scientific article; zbMATH DE number 472963 (Why is no real title available?)
• scientific article; zbMATH DE number 720689 (Why is no real title available?)
• scientific article; zbMATH DE number 3277086 (Why is no real title available?)
• Identification and Inference in Nonlinear Difference-in-Differences Models
• Improving the power of tests of stochastic dominance
• Non-Parametric Estimation. I. Validation of Order Statistics
• Nonparametric estimation and inference on conditional quantile processes
• Nonparametric Likelihood Confidence Bands for a Distribution Function
• On the exact Berk-Jones statistics and their p-value calculation
• Order Statistics
• Overweight tails are inefficient
• Putting Behavioral Economics to Work: Testing for Gift Exchange in Labor Markets Using Field Experiments
• Table for Estimating the Goodness of Fit of Empirical Distributions
• Testing for restricted stochastic dominance
• Testing Statistical Hypotheses
• The asymptotic distribution of the suprema of the standardized empirical processes
• The asymptotic distribution of the supremum of the standardized empirical distribution function on subintervals
• The Power to See: A New Graphical Test of Normality

Cited in

(18)

• Comparing a large number of multivariate distributions
• Comparison of multivariate distributions using quantile-quantile plots and related tests
• Some practical reasons for reconsidering the Kolmogorov‐Smirnov test
• How rare are power-law networks really?
• Fast calculation of p-values for one-sided Kolmogorov-Smirnov type statistics
• The distance between: an algorithmic approach to comparing stochastic models to time-series data
• Multiple Testing and the Distributional Effects of Accountability Incentives in Education
• Inference on Consensus Ranking of Distributions
• Correcting for Endogeneity in Models with Bunching
• ODC and ROC curves, comparison curves and stochastic dominance
• Exact and randomization distributions of Kolmogorov-Smirnov tests two or three samples
• Multiple testing of a function's monotonicity
• Uniform mean estimation for monotonic processes
• Finding exclusive peak with permutation testing
• Testing for homogeneous treatment effects in linear and nonparametric instrumental variable models
• Modelling monthly rainfall in the case of excessive zeros
• Pitman efficiency lower bounds for multivariate distribution-free tests based on optimal transport
• Permutation test for heterogeneous treatment effects with a nuisance parameter

Describes a project that uses

Uses Software

• R

This page was built for publication: Comparing distributions by multiple testing across quantiles or CDF values

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q1668577)