Precise Error Analysis of Regularized <inline-formula> <tex-math notation="LaTeX">M </tex-math> </inline-formula>-Estimators in High Dimensions
From MaRDI portal
Publication:4682865
Abstract: A popular approach for estimating an unknown signal from noisy, linear measurements is via solving a so called emph{regularized M-estimator}, which minimizes a weighted combination of a convex loss function and of a convex (typically, non-smooth) regularizer. We accurately predict the squared error performance of such estimators in the high-dimensional proportional regime. The random measurement matrix is assumed to have entries iid Gaussian, only minimal and rather mild regularity conditions are imposed on the loss function, the regularizer, and on the noise and signal distributions. We show that the error converges in probability to a nontrivial limit that is given as the solution to a minimax convex-concave optimization problem on four scalar optimization variables. We identify a new summary parameter, termed the Expected Moreau envelope to play a central role in the error characterization. The emph{precise} nature of the results permits an accurate performance comparison between different instances of regularized M-estimators and allows to optimally tune the involved parameters (e.g. regularizer parameter, number of measurements). The key ingredient of our proof is the emph{Convex Gaussian Min-max Theorem} (CGMT) which is a tight and strengthened version of a classical Gaussian comparison inequality that was proved by Gordon in 1988.
Cited in
(62)- Overcoming the limitations of phase transition by higher order analysis of regularization techniques
- The distribution of the Lasso: uniform control over sparse balls and adaptive parameter tuning
- Precise statistical analysis of classification accuracies for adversarial training
- Asymptotic normality of robust M-estimators with convex penalty
- Fundamental barriers to high-dimensional regression with convex penalties
- The asymptotic distribution of the MLE in high-dimensional logistic models: arbitrary covariance
- A precise high-dimensional asymptotic theory for boosting and minimum-\(\ell_1\)-norm interpolated classifiers
- Finite-sample analysis of \(M\)-estimators using self-concordance
- The likelihood ratio test in high-dimensional logistic regression is asymptotically a rescaled Chi-square
- Concentration of the Frobenius norm of generalized matrix inverses
- Learning curves of generic features maps for realistic datasets with a teacher-student model*
- Generalization error rates in kernel regression: the crossover from the noiseless to noisy regime*
- Binary classification of Gaussian mixtures: abundance of support vectors, benign overfitting, and regularization
- A Unifying Tutorial on Approximate Message Passing
- Sharp global convergence guarantees for iterative nonconvex optimization with random data
- Dimension Free Nonasymptotic Bounds on the Accuracy of High-Dimensional Laplace Approximation
- Automatic bias correction for testing in high‐dimensional linear models
- Debiasing convex regularized estimators and interval estimation in linear models
- LASSO Reloaded: A Variational Analysis Perspective with Applications to Compressed Sensing
- Noisy linear inverse problems under convex constraints: exact risk asymptotics in high dimensions
- Universality of regularized regression estimators in high dimensions
- The Lasso with general Gaussian designs with applications to hypothesis testing
- Tractability from overparametrization: the example of the negative perceptron
- A tradeoff between false discovery and true positive proportions for sparse high-dimensional logistic regression
- The curse of overparametrization in adversarial training: precise analysis of robust generalization for random features regression
- An error estimator for separated representations of highly multidimensional models
- Fluctuations, bias, variance and ensemble of learners: exact asymptotics for convex losses in high-dimension
- Phase transition and higher order analysis of L_q regularization under dependence
- Kronecker-product random matrices and a matrix least squares problem
- Performance of Bayesian linear regression in a model with mismatch
- Corrected generalized cross-validation for finite ensembles of penalized estimators
- Correlation adjusted debiased Lasso: debiasing the Lasso with inaccurate covariate model
- Entrywise dynamics and universality of general first order methods
- A leave-one-out approach to approximate message passing
- High-dimensional learning of narrow neural networks
- Exact threshold for approximate ellipsoid fitting of random points
- The generalization error of max-margin linear classifiers: benign overfitting and high dimensional asymptotics in the overparametrized regime
- A new central limit theorem for the augmented IPW estimator: variance inflation, cross-fit covariance and beyond
- Observable adjustments in single-index models for regularized M-estimators with bounded p/n
- Error estimation and adaptive tuning for unregularized robust M-estimator
- Injectivity of ReLU networks: perspectives from statistical physics
- Equivalence of state equations from different methods in high-dimensional regression
- High-dimensional asymptotics of denoising autoencoders
- Benign overfitting of non-sparse high-dimensional linear regression with correlated noise
- Dimension free ridge regression
- Precise asymptotics of bagging regularized M-estimators
- High-dimensional robust regression under heavy-tailed data: asymptotics and universality
- Analysis of high-dimensional Gaussian labeled-unlabeled mixture model via message-passing algorithm
- Differentially private learning beyond the classical dimensionality regime
- Bayes-optimal learning of deep random networks of extensive-width
- Asymptotic dynamics of alternating minimization for bilinear regression
- High-dimensional asymptotics of VAEs: threshold of posterior collapse and dataset-size dependence of rate-distortion curve
- Gaussian random projections of convex cones: approximate kinematic formulae and applications
- Universality of estimators for high-dimensional linear models with block dependency
- Gradient descent inference in empirical risk minimization
- Uncertainty quantification for iterative algorithms in linear models with application to early stopping
- Regularization using synthetic data for high-dimensional inference
- Few-shot multi-task learning of linear invariant features with meta subspace pursuit
- A nonasymptotic distributional theory of approximate message passing for sparse and robust regression
- Asymptotics of resampling without replacement in robust and logistic regression
- Moment-based adjustments of statistical inference in high-dimensional generalized linear models
- The nuclear route: sharp asymptotics of ERM in overparameterized quadratic networks
This page was built for publication: Precise Error Analysis of Regularized <inline-formula> <tex-math notation="LaTeX">$M$ </tex-math> </inline-formula>-Estimators in High Dimensions
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q4682865)
