A modern maximum-likelihood theory for high-dimensional logistic regression

DOI10.1073/PNAS.1810420116zbMATH Open1431.62084OpenAlexW2963060833WikidataQ91526601 ScholiaQ91526601MaRDI QIDQ5218552FDOQ5218552

Authors: Pragya Sur, Emmanuel J. Candès 

Publication date: 4 March 2020

Published in: Proceedings of the National Academy of Sciences (Search for Journal in Brave)

Abstract: Every student in statistics or data science learns early on that when the sample size largely exceeds the number of variables, fitting a logistic model produces estimates that are approximately unbiased. Every student also learns that there are formulas to predict the variability of these estimates which are used for the purpose of statistical inference; for instance, to produce p-values for testing the significance of regression coefficients. Although these formulas come from large sample asymptotics, we are often told that we are on reasonably safe grounds when $n$ is large in such a way that $nge5p$ or $nge10p$. This paper shows that this is far from the case, and consequently, inferences routinely produced by common software packages are often unreliable. Consider a logistic model with independent features in which $n$ and $p$ become increasingly large in a fixed ratio. Then we show that (1) the MLE is biased, (2) the variability of the MLE is far greater than classically predicted, and (3) the commonly used likelihood-ratio test (LRT) is not distributed as a chi-square. The bias of the MLE is extremely problematic as it yields completely wrong predictions for the probability of a case based on observed values of the covariates. We develop a new theory, which asymptotically predicts (1) the bias of the MLE, (2) the variability of the MLE, and (3) the distribution of the LRT. We empirically also demonstrate that these predictions are extremely accurate in finite samples. Further, an appealing feature is that these novel predictions depend on the unknown sequence of regression coefficients only through a single scalar, the overall strength of the signal. This suggests very concrete procedures to adjust inference; we describe one such procedure learning a single parameter from data and producing accurate inference

Full work available at URL: https://arxiv.org/abs/1803.06964

Recommendations

• Maximum likelihood estimation in logistic regression models with a diverging number of covariates
• The likelihood ratio test in high-dimensional logistic regression is asymptotically a rescaled Chi-square
• The phase transition for the existence of the maximum likelihood estimate in high-dimensional logistic regression
• Accuracies in the theory of logistic models
• Rate of convergence of the probability of non-existence of the MLE's in simple logistic regression

Mathematics Subject Classification ID

Point estimation (62F10) Asymptotic properties of parametric estimators (62F12) Generalized linear models (logistic models) (62J12)

Cited In (64)

• FDR control and power analysis for high-dimensional logistic regression via Stabkoff
• Bounded-memory adjusted scores estimation in generalized linear models with large data sets
• A comprehensive review of bias reduction methods for logistic regression
• Dimension-agnostic inference using cross U-statistics
• Noisy linear inverse problems under convex constraints: exact risk asymptotics in high dimensions
• StarTrek: combinatorial variable selection with false discovery rate control
• Tractability from overparametrization: the example of the negative perceptron
• Exact convergence analysis for metropolis–hastings independence samplers in Wasserstein distances
• Approximate message passing with rigorous guarantees for pooled data and quantitative group testing
• A tradeoff between false discovery and true positive proportions for sparse high-dimensional logistic regression
• On the functional regression model and its finite-dimensional approximations
• An adaptively resized parametric bootstrap for inference in high-dimensional generalized linear models
• A discussion of ``A note on universal inference by Tse and Davison
• A regularization-based adaptive test for high-dimensional GLMs
• Approximate message passing with spectral initialization for generalized linear models*
• A Scale-Free Approach for False Discovery Rate Control in Generalized Linear Models
• Hierarchical inference for genome-wide association studies: a view on methodology with software
• Title not available (Why is that?)
• Finite-sample analysis of \(M\)-estimators using self-concordance
• A Unifying Tutorial on Approximate Message Passing
• Approximate message passing algorithms for rotationally invariant matrices
• Discussion on: “A Scale-Free Approach for False Discovery Rate Control in Generalized Linear Models” by Dai, Lin, Zing, Liu
• Binary classification of Gaussian mixtures: abundance of support vectors, benign overfitting, and regularization
• The likelihood ratio test in high-dimensional logistic regression is asymptotically a rescaled Chi-square
• Consistency of logistic regression coefficient estimates calculated from a training sample.
• Sharp global convergence guarantees for iterative nonconvex optimization with random data
• Conjugate priors and bias reduction for logistic regression models
• Debiased lasso for generalized linear models with a diverging number of covariates
• The asymptotic distribution of the MLE in high-dimensional logistic models: arbitrary covariance
• Penalization-induced shrinking without rotation in high dimensional GLM regression: a cavity analysis
• Title not available (Why is that?)
• Title not available (Why is that?)
• Probabilistic learning inference of boundary value problem with uncertainties based on Kullback-Leibler divergence under implicit constraints
• Title not available (Why is that?)
• Maximum likelihood estimation in logistic regression models with a diverging number of covariates
• Universality of regularized regression estimators in high dimensions
• Automatic bias correction for testing in high‐dimensional linear models
• Mallows criterion for heteroskedastic linear regressions with many regressors
• Which bridge estimator is the best for variable selection?
• Precise statistical analysis of classification accuracies for adversarial training
• Statistical Inference for High-Dimensional Generalized Linear Models With Binary Outcomes
• The existence of maximum likelihood estimate in high-dimensional binary response generalized linear models
• Multicarving for high-dimensional post-selection inference
• Title not available (Why is that?)
• Online inference in high-dimensional generalized linear models with streaming data
• The phase transition for the existence of the maximum likelihood estimate in high-dimensional logistic regression
• Assessing the Most Vulnerable Subgroup to Type II Diabetes Associated with Statin Usage: Evidence from Electronic Health Record Data
• Comment on “A Scale-Free Approach for False Discovery Rate Control in Generalized Linear Models” by Chengguang Dai, Buyu Lin, Xin Xing, and Jun S. Liu
• Comments on “A Scale-Free Approach for False Discovery Rate Control in Generalized Linear Models”
• Directional testing for high dimensional multivariate normal distributions
• The distribution of the Lasso: uniform control over sparse balls and adaptive parameter tuning
• Title not available (Why is that?)
• Analysis of overfitting in the regularized Cox model
• Knockoffs with side information
• Some perspectives on inference in high dimensions
• Debiasing convex regularized estimators and interval estimation in linear models
• Replica analysis of overfitting in generalized linear regression models
• Global and Simultaneous Hypothesis Testing for High-Dimensional Logistic Regression Models
• Optimal combination of linear and spectral estimators for generalized linear models
• Universality of approximate message passing with semirandom matrices
• The Lasso with general Gaussian designs with applications to hypothesis testing
• Fundamental barriers to high-dimensional regression with convex penalties
• A precise high-dimensional asymptotic theory for boosting and minimum-\(\ell_1\)-norm interpolated classifiers
• A Friendly Tutorial on Mean-Field Spin Glass Techniques for Non-Physicists

This page was built for publication: A modern maximum-likelihood theory for high-dimensional logistic regression

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5218552)