WONDER: weighted one-shot distributed ridge regression in high dimensions
From MaRDI portal
(Redirected from Publication:4969115)
Abstract: In many areas, practitioners need to analyze large datasets that challenge conventional single-machine computing. To scale up data analysis, distributed and parallel computing approaches are increasingly needed. Here we study a fundamental and highly important problem in this area: How to do ridge regression in a distributed computing environment? Ridge regression is an extremely popular method for supervised learning, and has several optimality properties, thus it is important to study. We study one-shot methods that construct weighted combinations of ridge regression estimators computed on each machine. By analyzing the mean squared error in a high dimensional random-effects model where each predictor has a small effect, we discover several new phenomena. 1. Infinite-worker limit: The distributed estimator works well for very large numbers of machines, a phenomenon we call "infinite-worker limit". 2. Optimal weights: The optimal weights for combining local estimators sum to more than unity, due to the downward bias of ridge. Thus, all averaging methods are suboptimal. We also propose a new Weighted ONe-shot DistributEd Ridge regression (WONDER) algorithm. We test WONDER in simulation studies and using the Million Song Dataset as an example. There it can save at least 100x in computation time, while nearly preserving test accuracy.
Recommendations
Cites work
- A Deterministic Equivalent for the Analysis of Correlated MIMO Multiple Access Channels
- A partially linear framework for massive heterogeneous data
- A split-and-conquer approach for analysis of
- Algorithmic aspects of parallel data processing
- Communication lower bounds for statistical estimation problems via a distributed data processing inequality
- Communication-efficient algorithms for statistical optimization
- Communication-efficient distributed statistical inference
- Communication-efficient sparse regression
- Computational Limits of A Distributed Algorithm For Smoothing Spline
- Deterministic equivalents for certain functionals of large random matrices
- Distributed estimation of principal eigenspaces
- Distributed inference for linear support vector machine
- Distributed inference for quantile regression processes
- Distributed learning with regularized least squares
- Distributed optimization and statistical learning via the alternating direction method of multipliers
- Distributed testing and estimation under sparse high dimensional models
- Divide and conquer in nonstandard problems and the super-efficiency phenomenon
- Divide and conquer kernel ridge regression: a distributed algorithm with minimax optimal rates
- Flexible results for quadratic forms with applications to variance components estimation
- Free Random Variables
- High-dimensional asymptotics of prediction: ridge regression and classification
- scientific article; zbMATH DE number 996442 (Why is no real title available?)
- scientific article; zbMATH DE number 51132 (Why is no real title available?)
- scientific article; zbMATH DE number 1964693 (Why is no real title available?)
- scientific article; zbMATH DE number 6982986 (Why is no real title available?)
- scientific article; zbMATH DE number 3244317 (Why is no real title available?)
- Large sample covariance matrices and high-dimensional data analysis
- Learning theory of distributed regression with bias corrected regularization kernel network
- Lectures on the Combinatorics of Free Probability
- On high-dimensional misspecified mixed model analysis in genome-wide association study
- On the optimality of averaging in distributed statistical learning
- Parallel Programming
- Quantile regression under memory constraint
- Random matrix methods for wireless communications.
- Random matrix theory in statistics: a review
- REML estimation: Asymptotic behavior and related topics
- Ridge regression and asymptotic minimax estimation over spheres of growing dimension
- Spectral analysis of large dimensional random matrices
- Spectral convergence for a general class of random matrices
- Variance estimation in high-dimensional linear models
Cited in
(18)- Distributed linear regression by averaging
- Canonical thresholding for nonsparse high-dimensional linear regression
- WONDER
- What causes the test error? Going beyond bias-variance via ANOVA
- scientific article; zbMATH DE number 7415114 (Why is no real title available?)
- Distributed estimation with empirical likelihood
- High-Dimensional Analysis of Double Descent for Linear Regression with Random Projections
- Estimation and inference in sparse multivariate regression and conditional Gaussian graphical models under an unbalanced distributed setting
- Distributed estimation and inference for semiparametric binary response models
- MANDERA: malicious node detection in federated learning via ranking
- Distributed estimation of spiked eigenvalues in spiked population models
- Precise high-dimensional asymptotics for quantifying heterogeneous transfers
- Adaptive distributed kernel ridge regression: a feasible distributed learning scheme for data silos
- Doubly robust estimation with graphical structure among predictors for integrating probability and non-probability samples
- Distributed Mallows model averaging for ridge regressions
- Precise asymptotics of bagging regularized M-estimators
- One-shot distributed shrinkage estimation in sparse linear regression models
- Distributed sparse learning with high-dimensional and non-randomly stored data
This page was built for publication: WONDER: weighted one-shot distributed ridge regression in high dimensions
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q4969115)
