Statistical inference for model parameters in stochastic gradient descent
Let \(x^*\in\mathbb{R}^d\) be the true parameter of a statistical model. In common models, \(x^*\) is the minimizer of a convex objective function \(F\), i.e. \[ x^*=\operatorname{argmin}F(x),\quad F(x):=\mathbb{E} f(x,\zeta), \] where \(\zeta\) is the random sample from a probability distribution \(\Pi\) and \( f(x,\zeta)\) is the loss function. A popular optimization method for minimizing \(F\) is the stochastic gradient descent (SGD). Let \(x_0\) denote any given starting point. SGD is an iterative algorithm, where the \(i\)th iterate takes the form \[ x_i=x_{i-1}-\eta_i\nabla f(x_{i-1}, \zeta_i). \] The step size \(\eta_i\) is a decreasing nonrandom sequence, \(\zeta_i\) is the \(i\)th sample randomly drawn from \(\Pi\), and \(\nabla f(x, \zeta_i)\) denotes the gradient of \( f(x, \zeta_i)\) w.r.t. \(x.\) The algorithm outputs either the last iterate \(x_n,\) or the average iterate \(\bar{x}_n=n^{-1}\sum_{i=1}^n x_i.\) In the paper under review, \(f(\cdot, \zeta)\) is strictly convex and satisfies certain smoothness conditions. First, for fixed \(d,\) two consistent estimators of the asymptotic covariance of \(\bar{x}_n\) are proposed: (a) a plug-in estimator, and (b) a batch-means estimator, the latter only uses the iterates from SGD. Both estimators allow to construct asymptotic confidence intervals for \(x^*\) and hypothesis tests. Second, in high-dimensional sparse linear regression, where \(d\) can be much larger than the sample size \(n,\) the authors use a version of the SGD algorithm to minimize \(F(x)+\lambda \|x\|_1\), \(\lambda > 0,\) and construct a debiased estimator of each regression coefficient that is asymptotically normal. This gives a one-pass algorithm for computing both the sparse regression coefficients and confidence intervals, which is applicable to online data.
- Online bootstrap confidence intervals for the stochastic gradient descent estimator
- Scalable statistical inference for averaged implicit stochastic gradient descent
- Asymptotic and finite-sample properties of estimators based on stochastic gradients
- Scalable estimation strategies based on stochastic approximations: classical results and new insights
- Adaptive sampling for incremental optimization using stochastic gradient descent
- \(p\)-values for high-dimensional regression
- A general theory of hypothesis tests and confidence regions for sparse high dimensional models
- A proximal stochastic gradient method with progressive variance reduction
- A Stochastic Approximation Method
- Acceleration of Stochastic Approximation by Averaging
- Asymptotic and finite-sample properties of estimators based on stochastic gradients
- Batch means and spectral variance estimators in Markov chain Monte Carlo
- Confidence Intervals and Hypothesis Testing for High-Dimensional Regression
- Confidence intervals for low dimensional parameters in high dimensional linear models
- Confidence level solutions for stochastic programming
- Dual averaging methods for regularized stochastic learning and online optimization
- Estimating the asymptotic variance with batch means
- Fast global convergence of gradient methods for high-dimensional statistical recovery
- Fixed-Width Output Analysis for Markov Chain Monte Carlo
- High-dimensional graphs and variable selection with the Lasso
- High-dimensional variable screening and bias in subsequent inference, with an empirical comparison
- scientific article; zbMATH DE number 854710 (Why is no real title available?)
- Introduction to uncertainty quantification
- Least squares after model selection in high-dimensional sparse models
- On Asymptotic Normality in Stochastic Approximation
- On asymptotically optimal confidence regions and tests for high-dimensional models
- Optimal Stochastic Approximation Algorithms for Strongly Convex Stochastic Composite Optimization I: A Generic Algorithmic Framework
- Robust Stochastic Approximation Approach to Stochastic Programming
- Sharp Thresholds for High-Dimensional and Noisy Sparsity Recovery Using $\ell _{1}$-Constrained Quadratic Programming (Lasso)
- Simulation Output Analysis Using Standardized Time Series
- Statistical inference for model parameters in stochastic gradient descent
- Statistics for high-dimensional data. Methods, theory and applications.
- Strong Consistency and Other Properties of the Spectral Variance Estimator
- Sure independence screening for ultrahigh dimensional feature space. With discussion and authors' reply
- Online statistical inference for parameters estimation with linear-equality constraints
- Parameter calibration in wake effect simulation model with stochastic gradient descent and stratified sampling
- Statistical inference for model parameters in stochastic gradient descent
- Bridging the gap between constant step size stochastic gradient descent and Markov chains
- Variance comparison between infinitesimal perturbation analysis and likelihood ratio estimators to stochastic gradient
- Scalable statistical inference for averaged implicit stochastic gradient descent
- Online bootstrap confidence intervals for the stochastic gradient descent estimator
- scientific article; zbMATH DE number 6860839 (Why is no real title available?)
- Statistics of robust optimization: a generalized empirical likelihood approach
- Neural ODEs as the deep limit of ResNets with constant weights
- Variance reduced median-of-means estimator for Byzantine-robust distributed inference
- Online stochastic gradient descent on non-convex losses from high-dimensional inference
- Statistical inference for online decision making via stochastic gradient descent
- One-dimensional system arising in stochastic gradient descent
- scientific article; zbMATH DE number 7625199 (Why is no real title available?)
- Convergence acceleration of ensemble Kalman inversion in nonlinear settings
- Asymptotic analysis via stochastic differential equations of gradient descent algorithms in statistical and computational paradigms
- Statistical inference for the population landscape via moment-adjusted stochastic gradients
- Scalable estimation strategies based on stochastic approximations: classical results and new insights
- A probability approximation framework: Markov process approach
- Online Covariance Matrix Estimation in Stochastic Gradient Descent
- First-Order Newton-Type Estimator for Distributed Estimation and Inference
- Two-stage communication-efficient distributed sparse M-estimation with missing data
- An Asymptotic Analysis of Random Partition Based Minibatch Momentum Methods for Linear Regression Models
- Online Bootstrap Inference For Policy Evaluation In Reinforcement Learning
- Estimation and inference by stochastic optimization
- Confidence region for distributed stochastic optimization problem via stochastic gradient tracking method
- A selective review on statistical methods for massive data computation: distributed computing, subsampling, and minibatch techniques
- Online bootstrap inference for the geometric median
- Online Statistical Inference for Stochastic Optimization via Kiefer-Wolfowitz Methods
- Principles of statistical inference in online problems
- Scalable inference in functional linear regression with streaming data
- Local limit theorems and strong approximations for Robbins-Monro procedures
- Online estimation and inference for robust policy evaluation in reinforcement learning
- Online statistical inference in decision-making with matrix context
- When Composite Likelihood meets Stochastic Approximation
- Online inference in high-dimensional regression with streaming clustered data
- Statistical inference for smoothed quantile regression with streaming data
- Fast inference for quantile regression with tens of millions of observations
- Statistical inference of constrained stochastic optimization via sketched sequential quadratic programming
- Stochastic approximation based confidence regions for stochastic variational inequalities
- Online robust estimation and bootstrap inference for function-on-scalar regression
- Statistical inference for decentralized federated learning
- Scalable inference for nonparametric stochastic approximation in reproducing kernel Hilbert spaces
- Statistical inference for online decision making with Lasso loss function: in a contextual multi-armed bandit setting
- Gradient descent inference in empirical risk minimization
- Partial-overlapping batch means estimators for Markov chain Monte Carlo
- Online covariance matrix estimation in sketched Newton methods
- Error estimates between SGD with momentum and underdamped Langevin diffusion
- Renewable ℓ 1 -Regularized Linear Support Vector Machine with High-Dimensional Streaming Data
This page was built for publication: Statistical inference for model parameters in stochastic gradient descent
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2176618)
