Statistical inference for model parameters in stochastic gradient descent
From MaRDI portal
Abstract: The stochastic gradient descent (SGD) algorithm has been widely used in statistical estimation for large-scale data due to its computational and memory efficiency. While most existing works focus on the convergence of the objective function or the error of the obtained solution, we investigate the problem of statistical inference of true model parameters based on SGD when the population loss function is strongly convex and satisfies certain smoothness conditions. Our main contributions are two-fold. First, in the fixed dimension setup, we propose two consistent estimators of the asymptotic covariance of the average iterate from SGD: (1) a plug-in estimator, and (2) a batch-means estimator, which is computationally more efficient and only uses the iterates from SGD. Both proposed estimators allow us to construct asymptotically exact confidence intervals and hypothesis tests. Second, for high-dimensional linear regression, using a variant of the SGD algorithm, we construct a debiased estimator of each regression coefficient that is asymptotically normal. This gives a one-pass algorithm for computing both the sparse regression coefficients and confidence intervals, which is computationally attractive and applicable to online data.
Recommendations
- Online bootstrap confidence intervals for the stochastic gradient descent estimator
- Scalable statistical inference for averaged implicit stochastic gradient descent
- Asymptotic and finite-sample properties of estimators based on stochastic gradients
- Scalable estimation strategies based on stochastic approximations: classical results and new insights
- Adaptive sampling for incremental optimization using stochastic gradient descent
Cites work
- scientific article; zbMATH DE number 854710 (Why is no real title available?)
- A Stochastic Approximation Method
- A general theory of hypothesis tests and confidence regions for sparse high dimensional models
- A proximal stochastic gradient method with progressive variance reduction
- Acceleration of Stochastic Approximation by Averaging
- Asymptotic and finite-sample properties of estimators based on stochastic gradients
- Batch means and spectral variance estimators in Markov chain Monte Carlo
- Confidence Intervals and Hypothesis Testing for High-Dimensional Regression
- Confidence intervals for low dimensional parameters in high dimensional linear models
- Confidence level solutions for stochastic programming
- Dual averaging methods for regularized stochastic learning and online optimization
- Estimating the asymptotic variance with batch means
- Fast global convergence of gradient methods for high-dimensional statistical recovery
- Fixed-Width Output Analysis for Markov Chain Monte Carlo
- High-dimensional graphs and variable selection with the Lasso
- High-dimensional variable screening and bias in subsequent inference, with an empirical comparison
- Introduction to uncertainty quantification
- Least squares after model selection in high-dimensional sparse models
- On Asymptotic Normality in Stochastic Approximation
- On asymptotically optimal confidence regions and tests for high-dimensional models
- Optimal Stochastic Approximation Algorithms for Strongly Convex Stochastic Composite Optimization I: A Generic Algorithmic Framework
- Robust Stochastic Approximation Approach to Stochastic Programming
- Sharp Thresholds for High-Dimensional and Noisy Sparsity Recovery Using $\ell _{1}$-Constrained Quadratic Programming (Lasso)
- Simulation Output Analysis Using Standardized Time Series
- Statistical inference for model parameters in stochastic gradient descent
- Statistics for high-dimensional data. Methods, theory and applications.
- Strong Consistency and Other Properties of the Spectral Variance Estimator
- Sure independence screening for ultrahigh dimensional feature space. With discussion and authors' reply
- \(p\)-values for high-dimensional regression
Cited in
(29)- A probability approximation framework: Markov process approach
- Convergence acceleration of ensemble Kalman inversion in nonlinear settings
- Statistics of robust optimization: a generalized empirical likelihood approach
- First-Order Newton-Type Estimator for Distributed Estimation and Inference
- Scalable statistical inference for averaged implicit stochastic gradient descent
- Statistical inference for the population landscape via moment-adjusted stochastic gradients
- Two-stage communication-efficient distributed sparse M-estimation with missing data
- scientific article; zbMATH DE number 7370601 (Why is no real title available?)
- Online bootstrap confidence intervals for the stochastic gradient descent estimator
- Scalable estimation strategies based on stochastic approximations: classical results and new insights
- Online statistical inference for parameters estimation with linear-equality constraints
- Statistical inference for model parameters in stochastic gradient descent
- An Asymptotic Analysis of Random Partition Based Minibatch Momentum Methods for Linear Regression Models
- Online Bootstrap Inference For Policy Evaluation In Reinforcement Learning
- One-dimensional system arising in stochastic gradient descent
- Estimation and inference by stochastic optimization
- A selective review on statistical methods for massive data computation: distributed computing, subsampling, and minibatch techniques
- Online bootstrap inference for the geometric median
- scientific article; zbMATH DE number 7306890 (Why is no real title available?)
- Neural ODEs as the deep limit of ResNets with constant weights
- Variance comparison between infinitesimal perturbation analysis and likelihood ratio estimators to stochastic gradient
- Parameter calibration in wake effect simulation model with stochastic gradient descent and stratified sampling
- Bridging the gap between constant step size stochastic gradient descent and Markov chains
- Online Statistical Inference for Stochastic Optimization via Kiefer-Wolfowitz Methods
- Online Covariance Matrix Estimation in Stochastic Gradient Descent
- Statistical inference for online decision making via stochastic gradient descent
- scientific article; zbMATH DE number 7625199 (Why is no real title available?)
- Confidence region for distributed stochastic optimization problem via stochastic gradient tracking method
- scientific article; zbMATH DE number 6860839 (Why is no real title available?)
This page was built for publication: Statistical inference for model parameters in stochastic gradient descent
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2176618)
