Subsampling and Jackknifing: A Practically Convenient Solution for Large Data Analysis With Limited Computational Resources
From MaRDI portal
Publication:6092959
Abstract: Modern statistical analysis often encounters datasets with large sizes. For these datasets, conventional estimation methods can hardly be used immediately because practitioners often suffer from limited computational resources. In most cases, they do not have powerful computational resources (e.g., Hadoop or Spark). How to practically analyze large datasets with limited computational resources then becomes a problem of great importance. To solve this problem, we propose here a novel subsampling-based method with jackknifing. The key idea is to treat the whole sample data as if they were the population. Then, multiple subsamples with greatly reduced sizes are obtained by the method of simple random sampling with replacement. It is remarkable that we do not recommend sampling methods without replacement because this would incur a significant cost for data processing on the hard drive. Such cost does not exist if the data are processed in memory. Because subsampled data have relatively small sizes, they can be comfortably read into computer memory as a whole and then processed easily. Based on subsampled datasets, jackknife-debiased estimators can be obtained for the target parameter. The resulting estimators are statistically consistent, with an extremely small bias. Finally, the jackknife-debiased estimators from different subsamples are averaged together to form the final estimator. We theoretically show that the final estimator is consistent and asymptotically normal. Its asymptotic statistical efficiency can be as good as that of the whole sample estimator under very mild conditions. The proposed method is simple enough to be easily implemented on most practical computer systems and thus should have very wide applicability.
Cites work
- A distributed one-step estimator
- A statistical perspective on algorithmic leveraging
- Communication-efficient algorithms for statistical optimization
- Communication-efficient distributed statistical inference
- Distributed one-step upgraded estimation for non-uniformly and non-randomly distributed data
- Faster least squares approximation
- scientific article; zbMATH DE number 3053501 (Why is no real title available?)
- Jackknife, bootstrap and other resampling methods in regression analysis
- Mathematical Statistics
- Microeconometrics
- More efficient estimation for logistic regression with optimal subsamples
- Optimal Distributed Subsampling for Maximum Quasi-Likelihood Estimators With Massive Data
- Optimal subsampling for large sample logistic regression
- Randomized Algorithms for Matrices and Data
- The jackknife estimate of variance
Cited in
(6)- A review on design inspired subsampling for big data
- A distance metric-based space-filling subsampling method for nonparametric models
- A selective review on statistical methods for massive data computation: distributed computing, subsampling, and minibatch techniques
- CluBear: a subsampling package for interactive statistical analysis with massive data on a single machine
- Supervised Stratified Subsampling for Predictive Analytics
- A novel approach of empirical likelihood with massive data
This page was built for publication: Subsampling and Jackknifing: A Practically Convenient Solution for Large Data Analysis With Limited Computational Resources
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6092959)
