Statistical inference for model parameters in stochastic gradient descent
From MaRDI portal
Abstract: The stochastic gradient descent (SGD) algorithm has been widely used in statistical estimation for large-scale data due to its computational and memory efficiency. While most existing works focus on the convergence of the objective function or the error of the obtained solution, we investigate the problem of statistical inference of true model parameters based on SGD when the population loss function is strongly convex and satisfies certain smoothness conditions. Our main contributions are two-fold. First, in the fixed dimension setup, we propose two consistent estimators of the asymptotic covariance of the average iterate from SGD: (1) a plug-in estimator, and (2) a batch-means estimator, which is computationally more efficient and only uses the iterates from SGD. Both proposed estimators allow us to construct asymptotically exact confidence intervals and hypothesis tests. Second, for high-dimensional linear regression, using a variant of the SGD algorithm, we construct a debiased estimator of each regression coefficient that is asymptotically normal. This gives a one-pass algorithm for computing both the sparse regression coefficients and confidence intervals, which is computationally attractive and applicable to online data.
Recommendations
- Online bootstrap confidence intervals for the stochastic gradient descent estimator
- Scalable statistical inference for averaged implicit stochastic gradient descent
- Asymptotic and finite-sample properties of estimators based on stochastic gradients
- Scalable estimation strategies based on stochastic approximations: classical results and new insights
- Adaptive sampling for incremental optimization using stochastic gradient descent
Cites work
- \(p\)-values for high-dimensional regression
- A general theory of hypothesis tests and confidence regions for sparse high dimensional models
- A proximal stochastic gradient method with progressive variance reduction
- A Stochastic Approximation Method
- Acceleration of Stochastic Approximation by Averaging
- Asymptotic and finite-sample properties of estimators based on stochastic gradients
- Batch means and spectral variance estimators in Markov chain Monte Carlo
- Confidence Intervals and Hypothesis Testing for High-Dimensional Regression
- Confidence intervals for low dimensional parameters in high dimensional linear models
- Confidence level solutions for stochastic programming
- Dual averaging methods for regularized stochastic learning and online optimization
- Estimating the asymptotic variance with batch means
- Fast global convergence of gradient methods for high-dimensional statistical recovery
- Fixed-Width Output Analysis for Markov Chain Monte Carlo
- High-dimensional graphs and variable selection with the Lasso
- High-dimensional variable screening and bias in subsequent inference, with an empirical comparison
- scientific article; zbMATH DE number 854710 (Why is no real title available?)
- Introduction to uncertainty quantification
- Least squares after model selection in high-dimensional sparse models
- On Asymptotic Normality in Stochastic Approximation
- On asymptotically optimal confidence regions and tests for high-dimensional models
- Optimal Stochastic Approximation Algorithms for Strongly Convex Stochastic Composite Optimization I: A Generic Algorithmic Framework
- Robust Stochastic Approximation Approach to Stochastic Programming
- Sharp Thresholds for High-Dimensional and Noisy Sparsity Recovery Using $\ell _{1}$-Constrained Quadratic Programming (Lasso)
- Simulation Output Analysis Using Standardized Time Series
- Statistical inference for model parameters in stochastic gradient descent
- Statistics for high-dimensional data. Methods, theory and applications.
- Strong Consistency and Other Properties of the Spectral Variance Estimator
- Sure independence screening for ultrahigh dimensional feature space. With discussion and authors' reply
Cited in
(49)- Online statistical inference for parameters estimation with linear-equality constraints
- Parameter calibration in wake effect simulation model with stochastic gradient descent and stratified sampling
- Statistical inference for model parameters in stochastic gradient descent
- Bridging the gap between constant step size stochastic gradient descent and Markov chains
- Variance comparison between infinitesimal perturbation analysis and likelihood ratio estimators to stochastic gradient
- Scalable statistical inference for averaged implicit stochastic gradient descent
- Online bootstrap confidence intervals for the stochastic gradient descent estimator
- scientific article; zbMATH DE number 6860839 (Why is no real title available?)
- Statistics of robust optimization: a generalized empirical likelihood approach
- Neural ODEs as the deep limit of ResNets with constant weights
- Variance reduced median-of-means estimator for Byzantine-robust distributed inference
- Online stochastic gradient descent on non-convex losses from high-dimensional inference
- Statistical inference for online decision making via stochastic gradient descent
- One-dimensional system arising in stochastic gradient descent
- scientific article; zbMATH DE number 7625199 (Why is no real title available?)
- Convergence acceleration of ensemble Kalman inversion in nonlinear settings
- Asymptotic analysis via stochastic differential equations of gradient descent algorithms in statistical and computational paradigms
- Statistical inference for the population landscape via moment-adjusted stochastic gradients
- Scalable estimation strategies based on stochastic approximations: classical results and new insights
- A probability approximation framework: Markov process approach
- Online Covariance Matrix Estimation in Stochastic Gradient Descent
- First-Order Newton-Type Estimator for Distributed Estimation and Inference
- Two-stage communication-efficient distributed sparse M-estimation with missing data
- An Asymptotic Analysis of Random Partition Based Minibatch Momentum Methods for Linear Regression Models
- Online Bootstrap Inference For Policy Evaluation In Reinforcement Learning
- Estimation and inference by stochastic optimization
- Confidence region for distributed stochastic optimization problem via stochastic gradient tracking method
- A selective review on statistical methods for massive data computation: distributed computing, subsampling, and minibatch techniques
- Online bootstrap inference for the geometric median
- Online Statistical Inference for Stochastic Optimization via Kiefer-Wolfowitz Methods
- Principles of statistical inference in online problems
- Scalable inference in functional linear regression with streaming data
- Local limit theorems and strong approximations for Robbins-Monro procedures
- Online estimation and inference for robust policy evaluation in reinforcement learning
- Online statistical inference in decision-making with matrix context
- When Composite Likelihood meets Stochastic Approximation
- Online inference in high-dimensional regression with streaming clustered data
- Statistical inference for smoothed quantile regression with streaming data
- Fast inference for quantile regression with tens of millions of observations
- Statistical inference of constrained stochastic optimization via sketched sequential quadratic programming
- Stochastic approximation based confidence regions for stochastic variational inequalities
- Online robust estimation and bootstrap inference for function-on-scalar regression
- Statistical inference for decentralized federated learning
- Scalable inference for nonparametric stochastic approximation in reproducing kernel Hilbert spaces
- Statistical inference for online decision making with Lasso loss function: in a contextual multi-armed bandit setting
- Gradient descent inference in empirical risk minimization
- Partial-overlapping batch means estimators for Markov chain Monte Carlo
- Online covariance matrix estimation in sketched Newton methods
- Error estimates between SGD with momentum and underdamped Langevin diffusion
This page was built for publication: Statistical inference for model parameters in stochastic gradient descent
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2176618)
