Online Covariance Matrix Estimation in Stochastic Gradient Descent
From MaRDI portal
Publication:6107216
Abstract: The stochastic gradient descent (SGD) algorithm is widely used for parameter estimation, especially for huge data sets and online learning. While this recursive algorithm is popular for computation and memory efficiency, quantifying variability and randomness of the solutions has been rarely studied. This paper aims at conducting statistical inference of SGD-based estimates in an online setting. In particular, we propose a fully online estimator for the covariance matrix of averaged SGD iterates (ASGD) only using the iterates from SGD. We formally establish our online estimator's consistency and show that the convergence rate is comparable to offline counterparts. Based on the classic asymptotic normality results of ASGD, we construct asymptotically valid confidence intervals for model parameters. Upon receiving new observations, we can quickly update the covariance matrix estimate and the confidence intervals. This approach fits in an online setting and takes full advantage of SGD: efficiency in computation and memory.
Recommendations
- Online estimation of the asymptotic variance for averaged stochastic gradient algorithms
- Online stochastic gradient descent on non-convex losses from high-dimensional inference
- Stochastic Gauss-Newton algorithms for online PCA
- Near-optimal stochastic approximation for online principal component estimation
- Online Variance Minimization
- Online variance minimization
- Statistical inference for online decision making via stochastic gradient descent
- Online regression with varying Gaussians and non-identical distributions
- Online kernel estimation of stationary stochastic diffusion models
- Online bootstrap confidence intervals for the stochastic gradient descent estimator
Cites work
- A nonstandard empirical likelihood for time series
- A Progressive Block Empirical Likelihood Method for Time Series
- A Stochastic Approximation Method
- Acceleration of Stochastic Approximation by Averaging
- Adaptive subgradient methods for online learning and stochastic optimization
- Analysis of recursive stochastic algorithms
- Approximation Methods which Converge with Probability one
- Asymptotic and finite-sample properties of estimators based on stochastic gradients
- Asymptotic Distribution of Stochastic Approximation Procedures
- Batch means and spectral variance estimators in Markov chain Monte Carlo
- Beyond the regret minimization barrier: optimal algorithms for stochastic strongly-convex optimization
- Computer-intensive rate estimation, diverging statistics and scanning
- Empirical likelihood methods with weakly dependent processes
- Estimating the asymptotic variance with batch means
- Fixed-Width Output Analysis for Markov Chain Monte Carlo
- scientific article; zbMATH DE number 3449561 (Why is no real title available?)
- Multivariate output analysis for Markov chain Monte Carlo
- On Asymptotic Normality in Stochastic Approximation
- On power series diverging everywhere on the circle of convergence
- Online bootstrap confidence intervals for the stochastic gradient descent estimator
- Online learning for matrix factorization and sparse coding
- Recursive estimation of time-average variance constants
- Resampling methods for dependent data
- Scalable statistical inference for averaged implicit stochastic gradient descent
- Statistical inference for model parameters in stochastic gradient descent
- Statistical inference for the population landscape via moment-adjusted stochastic gradients
- Stochastic approximation
- Stochastic Estimation of the Maximum of a Regression Function
- Subsampling
- Theoretical comparisons of block bootstrap methods
Cited in
(24)- scientific article; zbMATH DE number 5957285 (Why is no real title available?)
- A Stochastic Majorize-Minimize Subspace Algorithm for Online Penalized Least Squares Estimation
- scientific article; zbMATH DE number 7625199 (Why is no real title available?)
- An Asymptotic Analysis of Random Partition Based Minibatch Momentum Methods for Linear Regression Models
- Confidence region for distributed stochastic optimization problem via stochastic gradient tracking method
- Parameter-parallel distributed variational quantum algorithm
- A selective review on statistical methods for massive data computation: distributed computing, subsampling, and minibatch techniques
- Online Statistical Inference for Stochastic Optimization via Kiefer-Wolfowitz Methods
- Principles of statistical inference in online problems
- Online statistical inference in decision-making with matrix context
- When Composite Likelihood meets Stochastic Approximation
- Statistical inference for smoothed quantile regression with streaming data
- Fast inference for quantile regression with tens of millions of observations
- Statistical inference of constrained stochastic optimization via sketched sequential quadratic programming
- Efficient and provable online reduced rank regression via online gradient descent
- Stochastic approximation based confidence regions for stochastic variational inequalities
- Online debiased lasso estimation and inference for heterogenous updating regressions
- Statistical inference for decentralized federated learning
- Scalable inference for nonparametric stochastic approximation in reproducing kernel Hilbert spaces
- A renewable estimation method for partially linear regression models
- Gradient descent inference in empirical risk minimization
- Partial-overlapping batch means estimators for Markov chain Monte Carlo
- Online covariance matrix estimation in sketched Newton methods
- Renewable penalized linear regression via inverse probability weighting for streaming data with missing covariates
This page was built for publication: Online Covariance Matrix Estimation in Stochastic Gradient Descent
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6107216)
