An Asymptotic Analysis of Random Partition Based Minibatch Momentum Methods for Linear Regression Models

DOI10.1080/10618600.2022.2143786arXiv2111.01507MaRDI QIDQ6180738FDOQ6180738

Authors: Yuan Gao, Xuening Zhu, Haobo Qi, Guodong Li, Riquan Zhang, Hansheng Wang 

Publication date: 22 January 2024

Published in: Journal of Computational and Graphical Statistics (Search for Journal in Brave)

Abstract: Momentum methods have been shown to accelerate the convergence of the standard gradient descent algorithm in practice and theory. In particular, the minibatch-based gradient descent methods with momentum (MGDM) are widely used to solve large-scale optimization problems with massive datasets. Despite the success of the MGDM methods in practice, their theoretical properties are still underexplored. To this end, we investigate the theoretical properties of MGDM methods based on the linear regression models. We first study the numerical convergence properties of the MGDM algorithm and further provide the theoretically optimal tuning parameters specification to achieve faster convergence rate. In addition, we explore the relationship between the statistical properties of the resulting MGDM estimator and the tuning parameters. Based on these theoretical findings, we give the conditions for the resulting estimator to achieve the optimal statistical efficiency. Finally, extensive numerical experiments are conducted to verify our theoretical results.

Full work available at URL: https://arxiv.org/abs/2111.01507

zbMATH Keywords

gradient descentstatistical efficiencymomentum methodnumerical convergence ratefixed minibatchshuffled minibatch

Mathematics Subject Classification ID

Statistics (62-XX)

Cites Work

• A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems
• Linear Statistical Inference and its Applications
• One-step sparse estimates in nonconcave penalized likelihood models
• Regularized estimation of large covariance matrices
• Acceleration of Stochastic Approximation by Averaging
• High-dimensional statistics. A non-asymptotic viewpoint
• High-dimensional probability. An introduction with applications in data science
• A Stochastic Approximation Method
• Deep learning
• Mathematical Statistics
• A statistical perspective on algorithmic leveraging
• Learning Bounds for Kernel Regression Using Effective Data Dimensionality
• Title not available (Why is that?)
• Forward regression for ultra-high dimensional variable screening
• A proximal stochastic gradient method with progressive variance reduction
• A split-and-conquer approach for analysis of
• Some methods of speeding up the convergence of iteration methods
• Distributed inference for linear support vector machine
• Divide and conquer local average regression
• Lectures on convex optimization
• Optimization methods for large-scale machine learning
• Distributed testing and estimation under sparse high dimensional models
• Distributed semi-supervised learning with kernel ridge regression
• Asymptotic and finite-sample properties of estimators based on stochastic gradients
• Momentum and stochastic momentum for stochastic gradient, Newton, proximal point and subspace descent methods
• Distributed estimation of principal eigenspaces
• Quantile regression under memory constraint
• Statistical foundations of data science
• Renewable estimation and incremental inference in generalized linear models with streaming data sets
• Title not available (Why is that?)
• Statistical inference for model parameters in stochastic gradient descent
• Online bootstrap confidence intervals for the stochastic gradient descent estimator
• Distributed Estimation for Principal Component Analysis: An Enlarged Eigenspace Analysis
• Distributed simultaneous inference in generalized linear models via confidence distribution
• Online Covariance Matrix Estimation in Stochastic Gradient Descent

Cited In (1)

• A selective review on statistical methods for massive data computation: distributed computing, subsampling, and minibatch techniques

This page was built for publication: An Asymptotic Analysis of Random Partition Based Minibatch Momentum Methods for Linear Regression Models

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6180738)