The likelihood ratio test in high-dimensional logistic regression is asymptotically a rescaled Chi-square

DOI10.1007/S00440-018-00896-9MaRDI QIDQ2273603zbMATH OpenOpenAlexWikidataFDO

Authors Pragya Sur, Yuxin Chen, Emmanuel J. Candès

Publication date 24 September 2019

Published in Zeitschrift für Wahrscheinlichkeitstheorie und Verwandte Gebiete (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1706.01191

zbMATH Keywords

approximate message passingconcentration inequalitiesconvex geometrygoodness of fithigh-dimensionalityleave-one-out analysislikelihood-ratio testslogistic regressionWilks' theorem

Mathematics Subject Classification ID

Asymptotic distribution theory in statistics (62E20) Generalized linear models (logistic models) (62J12)

Abstract: Logistic regression is used thousands of times a day to fit data, predict future outcomes, and assess the statistical significance of explanatory variables. When used for the purpose of statistical inference, logistic models produce p-values for the regression coefficients by using an approximation to the distribution of the likelihood-ratio test. Indeed, Wilks' theorem asserts that whenever we have a fixed number $p$ of variables, twice the log-likelihood ratio (LLR) $2Lambda$ is distributed as a $ch{i}_k^2$ variable in the limit of large sample sizes $n$; here, $k$ is the number of variables being tested. In this paper, we prove that when $p$ is not negligible compared to $n$, Wilks' theorem does not hold and that the chi-square approximation is grossly incorrect; in fact, this approximation produces p-values that are far too small (under the null hypothesis). Assume that $n$ and $p$ grow large in such a way that $p/nightarrowkappa$ for some constant $kappa<1/2$. We prove that for a class of logistic models, the LLR converges to a rescaled chi-square, namely, $2Lambdastackrelmathrmdightarrowalpha(kappa)ch{i}_k^2$, where the scaling factor $alpha(kappa)$ is greater than one as soon as the dimensionality ratio $kappa$ is positive. Hence, the LLR is larger than classically assumed. For instance, when $kappa=0.3$, $alpha(kappa)approx1.5$. In general, we show how to compute the scaling factor by solving a nonlinear system of two equations with two unknowns. Our mathematical arguments are involved and use techniques from approximate message passing theory, non-asymptotic random matrix theory and convex geometry. We also complement our mathematical study by showing that the new limiting distribution is accurate for finite sample sizes. Finally, all the results from this paper extend to some other regression models such as the probit regression model.

Review text

The paper deals with a testing problem of binary regressions (e.g. the logistic model or the probit model), and concentrates on the distribution of twice the log-likelihood ratio (2LLR) under the null-hypothesis. The authors illustrate that the commonly used Wilks' theorem stating that the distribution of 2LLR will tend for increasing sample size \(n\) to a chi-square, does not hold when the number of variables included in the model is not negligible compared to \(n\). Instead, for a class of models, a rescaled chi-square has to be used to approximate the 2LLR distribution and the corresponding \(p\)-values adequately. The \(p\)-values obtained using the chi-square approximation without the rescaling factor are too small. The authors also show that the required rescaling factor can be obtained by solving a nonlinear system of two equations with two variables. The paper is well written and a very nice example of rigorous mathematical statistics. A large part of the paper is dedicated to proofs and thus it may be mainly of interest for researchers working in the field of mathematical statistics. People working with these models to analyze data might be satisfied with the take home messages.

Recommendations

• The likelihood ratio test for high-dimensional linear regression model
• Likelihood ratio tests under model misspecification in high dimensions
• Likelihood ratio test in multivariate linear regression: from low to high dimension
• Global and Simultaneous Hypothesis Testing for High-Dimensional Logistic Regression Models
• Asymptotic power of likelihood ratio tests for high dimensional data
• Likelihood ratio tests for high-dimensional normal distributions
• Empirical Likelihood Ratio Tests for Coefficients in High Dimensional Heteroscedastic Linear Models
• A note on asymptotics of classical likelihood ratio tests for high-dimensional normal distributions
• Empirical Likelihood Test for High Dimensional Generalized Linear Models

Cites work

• A Bound on Tail Probabilities for Quadratic Forms in Independent Random Variables
• A decomposition for the likelihood ratio statistic and the Bartlett correction - a Bayesian argument
• A GENERAL DISTRIBUTION THEORY FOR A CLASS OF LIKELIHOOD CRITERIA
• A GENERAL METHOD FOR APPROXIMATING TO THE DISTRIBUTION OF LIKELIHOOD RATIO CRITERIA
• A high-dimensional Wilks phenomenon
• A modern maximum-likelihood theory for high-dimensional logistic regression
• A tail inequality for quadratic forms of subgaussian random vectors
• An introduction to Bartlett correction and bias reduction
• Asymptotic behavior of likelihood methods for exponential families when the number of parameters tends to infinity
• Asymptotic behavior of M estimators of p regression parameters when \(p^ 2/n\) is large. II: Normal approximation
• Asymptotic behavior of M-estimators of p regression parameters when p^ 2/n is large. I. Consistency
• Asymptotic behavior of the empiric distribution of M-estimated residuals from a regression model with many parameters
• Asymptotic Statistics
• Asymptotics with increasing dimension for robust regression with applications to the bootstrap
• Bartlett correction factors in logistic regression models
• Bartlett corrections for one-parameter exponential family models
• Debiasing the Lasso: optimal sample size for Gaussian designs
• Enumeration of Seven-Argument Threshold Functions
• False discoveries occur early on the Lasso path
• Finding hidden cliques of size \(\sqrt{N/e}\) in nearly linear time
• Generalized likelihood ratio statistics and Wilks phenomenon
• Hanson-Wright inequality and sub-Gaussian concentration
• High dimensional robust M-estimation: asymptotic variance via approximate message passing
• High-dimensional generalized linear models and the lasso
• scientific article; zbMATH DE number 3841053 (Why is no real title available?)
• scientific article; zbMATH DE number 47310 (Why is no real title available?)
• scientific article; zbMATH DE number 107482 (Why is no real title available?)
• scientific article; zbMATH DE number 6026126 (Why is no real title available?)
• Living on the edge: phase transitions in convex programs with random data
• Maximum likelihood estimation in logistic regression models with a diverging number of covariates
• Mills' ratio: Monotonicity patterns and functional inequalities
• Nonconcave Penalized Likelihood With NP-Dimensionality
• Nonparametric inference with generalized likelihood ratio tests (With comments and rejoinder)
• Nonuniformity of p-values can occur early in diverging dimensions
• On asymptotically optimal confidence regions and tests for high-dimensional models
• On bartlett and bartlett-type corrections francisco cribari-neto
• On parameters of increasing dimensions
• On robust regression with high-dimensional predictors
• On the Distribution of the Likelihood Ratio
• On the impact of predictor geometry on the performance on high-dimensional ridge-regularized generalized robust regression estimators
• Panning for Gold: ‘Model-X’ Knockoffs for High Dimensional Controlled Variable Selection
• Penalized high-dimensional empirical likelihood
• Penalized maximum likelihood estimation and effective dimension
• Precise Error Analysis of Regularized <inline-formula> <tex-math notation="LaTeX">$M$ </tex-math> </inline-formula>-Estimators in High Dimensions
• Properties of sufficiency and statistical tests
• Robust regression: Asymptotics, conjectures and Monte Carlo
• Some Inequalities on Mill's Ratio and Related Functions
• Some remarks on latent variable models in categorical data analysis
• State evolution for general approximate message passing algorithms, with applications to spatial coupling
• Testing Statistical Hypotheses
• The Dynamics of Message Passing on Dense Graphs, with Applications to Compressed Sensing
• The LASSO Risk for Gaussian Matrices
• The likelihood ratio test in high-dimensional logistic regression is asymptotically a rescaled Chi-square
• The probabilistic method. With an appendix on the life and work of Paul Erdős.
• Universality in polytope phase transitions and message passing algorithms
• Universality laws for randomized dimension reduction, with applications
• Updating the Inverse of a Matrix

Cited in

(56)

• The generalized Lasso problem and uniqueness
• Scale calibration for high-dimensional robust regression
• Reproducible learning in large-scale graphical models
• Directional testing for high dimensional multivariate normal distributions
• Approximate message passing algorithms for rotationally invariant matrices
• The asymptotic distribution of the MLE in high-dimensional logistic models: arbitrary covariance
• Thresholding tests based on affine Lasso to achieve non-asymptotic nominal level and high power under sparse and dense alternatives in high dimension
• A precise high-dimensional asymptotic theory for boosting and minimum-\(\ell_1\)-norm interpolated classifiers
• Weighted Lasso estimates for sparse logistic regression: non-asymptotic properties with measurement errors
• The phase transition for the existence of the maximum likelihood estimate in high-dimensional logistic regression
• Implicit regularization in nonconvex statistical estimation: gradient descent converges linearly for phase retrieval, matrix completion, and blind deconvolution
• Bounds for the asymptotic distribution of the likelihood ratio
• The existence of maximum likelihood estimate in high-dimensional binary response generalized linear models
• Which bridge estimator is the best for variable selection?
• The likelihood ratio test in high-dimensional logistic regression is asymptotically a rescaled Chi-square
• Spectral method and regularized MLE are both optimal for top-\(K\) ranking
• Gradient descent with random initialization: fast global convergence for nonconvex phase retrieval
• Performance of the Likelihood Ratio Test When Fitting Logistic Regression Models with Small Samples
• Global and Simultaneous Hypothesis Testing for High-Dimensional Logistic Regression Models
• scientific article; zbMATH DE number 7626769 (Why is no real title available?)
• Noisy matrix completion: understanding statistical guarantees for convex relaxation via nonconvex optimization
• A modern maximum-likelihood theory for high-dimensional logistic regression
• A Unifying Tutorial on Approximate Message Passing
• Replica analysis of overfitting in generalized linear regression models
• Characterizing the SLOPE trade-off: a variational perspective and the Donoho-Tanner limit
• Automatic bias correction for testing in high‐dimensional linear models
• Discussion on: “A Scale-Free Approach for False Discovery Rate Control in Generalized Linear Models” by Dai, Lin, Zing, Liu
• Knockoffs with side information
• Statistical Inference for High-Dimensional Generalized Linear Models With Binary Outcomes
• Moderate-Dimensional Inferences on Quadratic Functionals in Ordinary Least Squares
• Universality of approximate message passing with semirandom matrices
• A tradeoff between false discovery and true positive proportions for sparse high-dimensional logistic regression
• A stable and adaptive polygenic signal detection method based on repeated sample splitting
• FDR control and power analysis for high-dimensional logistic regression via Stabkoff
• Universality of approximate message passing algorithms and tensor networks
• Comparing algorithms for characterizing treatment effect heterogeneity in randomized trials
• Markov neighborhood regression for statistical inference of high-dimensional generalized linear models
• Comparing methods to assess treatment effect heterogeneity in general parametric regression models
• Likelihood Ratio Tests in Random Graph Models with Increasing Dimensions
• Ah-knockoff: false discovery rate control in high-dimensional additive hazards models
• Entrywise dynamics and universality of general first order methods
• A leave-one-out approach to approximate message passing
• A new central limit theorem for the augmented IPW estimator: variance inflation, cross-fit covariance and beyond
• Observable adjustments in single-index models for regularized M-estimators with bounded p/n
• High dimensional logistic regression under network dependence
• Inference for possibly misspecified generalized linear models with nonpolynomial-dimensional nuisance parameters
• Spectrum-aware debiasing: a modern inference framework with applications to principal components regression
• Interpretable Model Summaries Using the Wasserstein Distance
• Dimension-free uniform concentration bound for logistic regression
• Universality of estimators for high-dimensional linear models with block dependency
• Gradient descent inference in empirical risk minimization
• Regularization using synthetic data for high-dimensional inference
• On the phase transition of Wilks' phenomenon under block compound symmetry covariance structure
• BinGFI: a fully automated framework for generalized fiducial inference for binary data
• A nonasymptotic distributional theory of approximate message passing for sparse and robust regression
• Moment-based adjustments of statistical inference in high-dimensional generalized linear models

This page was built for publication: The likelihood ratio test in high-dimensional logistic regression is asymptotically a rescaled Chi-square

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2273603)