On the optimality of averaging in distributed statistical learning
From MaRDI portal
Abstract: A common approach to statistical learning with big-data is to randomly split it among machines and learn the parameter of interest by averaging the individual estimates. In this paper, focusing on empirical risk minimization, or equivalently M-estimation, we study the statistical error incurred by this strategy. We consider two large-sample settings: First, a classical setting where the number of parameters is fixed, and the number of samples per machine . Second, a high-dimensional regime where both with . For both regimes and under suitable assumptions, we present asymptotically exact expressions for this estimation error. In the fixed- setting, under suitable assumptions, we prove that to leading order averaging is as accurate as the centralized solution. We also derive the second order error terms, and show that these can be non-negligible, notably for non-linear models. The high-dimensional setting, in contrast, exhibits a qualitatively different behavior: data splitting incurs a first-order accuracy loss, which to leading order increases linearly with the number of machines. The dependence of our error approximations on the number of machines traces an interesting accuracy-complexity tradeoff, allowing the practitioner an informed choice on the number of machines to deploy. Finally, we confirm our theoretical analysis with several simulations.
Recommendations
- Distributed linear regression by averaging
- Communication-efficient algorithms for statistical optimization
- Learning theory of distributed regression with bias corrected regularization kernel network
- Nonparametric distributed learning under general designs
- Distributed statistical optimization for non-randomly stored big data with application to penalized learning
Cites work
- Asymptotic Statistics
- Communication-efficient algorithms for statistical optimization
- Geometric median and robust estimation in Banach spaces
- scientific article; zbMATH DE number 1391247 (Why is no real title available?)
- scientific article; zbMATH DE number 1409619 (Why is no real title available?)
- Loss minimization and parameter estimation with heavy tails
- On robust regression with high-dimensional predictors
- Piecewise linear regularized solution paths
- Robust regression: Asymptotics, conjectures and Monte Carlo
- The price of bias reduction when there is no unbiased estimate
- The second-order bias and mean squared error of nonlinear estimators
- Understanding machine learning. From theory to algorithms
Cited in
(60)- Distributed testing and estimation under sparse high dimensional models
- A distributed one-step estimator
- Distributed linear regression by averaging
- Distributed adaptive Huber regression
- Distributed optimization and statistical learning for large-scale penalized expectile regression
- Robust distributed estimation and variable selection for massive datasets via rank regression
- Distributed kernel gradient descent algorithm for minimum error entropy principle
- Nonparametric distributed learning under general designs
- Adaptive distributed methods under communication constraints
- Robust communication-efficient distributed composite quantile regression and variable selection for massive data
- Distributed one-step upgraded estimation for non-uniformly and non-randomly distributed data
- Distributed statistical estimation and rates of convergence in normal approximation
- Averaging versus voting: a comparative study of strategies for distributed classification
- Distributed function estimation: adaptation using minimal communication
- Communication-efficient algorithms for statistical optimization
- Parallelizing stochastic gradient descent for least squares regression: mini-batching, averaging, and model misspecification
- Learning theory of distributed regression with bias corrected regularization kernel network
- WONDER: weighted one-shot distributed ridge regression in high dimensions
- Distributed Averaging Via Lifted Markov Chains
- Variance reduced median-of-means estimator for Byzantine-robust distributed inference
- scientific article; zbMATH DE number 7625155 (Why is no real title available?)
- Communication-Efficient Distributed Linear Discriminant Analysis for Binary Classification
- Communication-Efficient Distributed Eigenspace Estimation
- A review of distributed statistical inference
- Global debiased DC estimations for biased estimators via pro forma regression
- Multivariate survival analysis in big data: A divide‐and‐combine approach
- Model aggregation for doubly divided data with large size and large dimension
- Robust distributed multicategory angle-based classification for massive data
- A communication efficient distributed one-step estimation
- Design of distributed BFGS algorithm and large sample properties for parameter estimation
- Communication-efficient distributed estimation for high-dimensional large-scale linear regression
- LIC criterion for optimal subset selection in distributed interval estimation
- Communication-Efficient Accurate Statistical Estimation
- Distributed statistical optimization for non-randomly stored big data with application to penalized learning
- scientific article; zbMATH DE number 7800967 (Why is no real title available?)
- Distributed sequential estimation procedures
- Distributed robust regression with correntropy losses and regularization kernel networks
- Aggregated inference
- Distributed subsampling for multiplicative regression
- A selective review on statistical methods for massive data computation: distributed computing, subsampling, and minibatch techniques
- Distributed parameter estimation framework based on moment method
- Distributed robust estimation and inference with contaminated data
- Communication-efficient distributed composite quantile regression via convolution smoothing and Poisson subsampling
- The ``component debiasing method in distributed Byzantine problems
- Adaptive distributed kernel ridge regression: a feasible distributed learning scheme for data silos
- Distributed learning for high-dimensional factor augmented regression models
- Divide-and-conquer offline policy evaluation for contextual bandits
- Analyzing the dissemination of news by model averaging and subsampling
- Communication-efficient distributed gradient descent via random projection
- Communication-Efficient Nonparametric Quantile Regression via Random Features
- Distributed estimation for large-scale expectile regression
- A communication-efficient method for generalized linear regression with ℓ 0 regularization
- Efficient distributed optimization for large-scale high-dimensional sparse penalized Huber regression
- Selective inference with distributed data
- Unified algorithms for distributed regularized linear regression model
- Composite quantile regression for a distributed system with non-randomly distributed data
- Communication-Efficient Pilot Estimation for Non-Randomly Distributed Data in Diverging Dimensions
- Additive-effect assisted learning
- Byzantine-Robust Distributed One-Step Estimation
- Distributed Nonparametric Regression with Heterogeneity Through Prediction-Based Aggregation
This page was built for publication: On the optimality of averaging in distributed statistical learning
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q4606521)
