Local uncertainty sampling for large-scale multiclass logistic regression

DOI10.1214/19-AOS1867MaRDI QIDQ2196246zbMATH OpenOpenAlexFDO

Authors Lei Han, Kean Ming Tan, Ting Yang, Tong Zhang

Publication date 28 August 2020

Published in The Annals of Statistics (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1604.08098, https://projecteuclid.org/euclid.aos/1594972838

zbMATH Keywords

binary and multiclass logistic regressionlocal case control samplinglocal uncertainty sampling

Mathematics Subject Classification ID

Sampling theory, sample surveys (62D05) Point estimation (62F10) Generalized linear models (logistic models) (62J12)

Abstract: A major challenge for building statistical models in the big data era is that the available data volume far exceeds the computational capability. A common approach for solving this problem is to employ a subsampled dataset that can be handled by available computational resources. In this paper, we propose a general subsampling scheme for large-scale multi-class logistic regression and examine the variance of the resulting estimator. We show that asymptotically, the proposed method always achieves a smaller variance than that of the uniform random sampling. Moreover, when the classes are conditionally imbalanced, significant improvement over uniform sampling can be achieved. Empirical performance of the proposed method is compared to other methods on both simulated and real-world datasets, and these results match and confirm our theoretical analysis.

Review text

For analyzing huge data sets using multiclass logistic regression when computational facilities are not available, one of the often used methods is to subsample a data set which can be accommodated within the available computer resources. There are two types of imbalances in classes, namely marginal imbalance (MI) when some classes are rarer than others and conditional imbalance (CI) when the class labels are easy to predict for most of the observations. For MI binary classification, case control (CC) subsampling is used with an equal number of samples from each class uniformly. In this paper, the authors review one of the earlier subsampling schemes for a binary logistic regression termed as local case control (LCC) sampling. This scheme is shown to fare better than the uniform random sampling with respect to asymptotic variance criterion of the estimates obtained. Next, they propose general subsampling schemes for large scale multiclass logistic regression problems. The method consists of selecting data points with labels that are conditionally uncertain given their local observations based on the predicted probability distribution and then fitting a multiclass logistic model for estimating the model parameter. Simulation and real world data sets, namely MNIST and Web-spam data are considered and it is confirmed that the LUS method fares better than uniform sampling, CC sampling and LCC sampling under various settings. If the full sample size $(n)$ based mle has asymptotic variance $v$, then the LUS has asymptotic variance less than $e v$ $(e>1)$, now based on a sample size of $n/ e$.

Recommendations

• Optimal subsampling for large sample logistic regression
• Optimal subsample selection for massive logistic regression with distributed data
• Multiclass-penalized logistic regression
• Probability estimation for large-margin classifiers
• Multi-output local Gaussian process regression: applications to uncertainty quantification
• Efficient posterior sampling for high-dimensional imbalanced logistic regression
• Logistic Discrimination Based on Regularized Local Likelihood Method

Cites work

• A Generalization of Sampling Without Replacement From a Finite Universe
• Additive logistic regression: a statistical view of boosting. (With discussion and a rejoinder by the authors)
• Fitting Logistic Regression Models in Stratified Case-Control Studies
• scientific article; zbMATH DE number 3984372 (Why is no real title available?)
• Local case-control sampling: efficient subsampling in imbalanced data sets
• On the Robustness of Weighted Methods for Fitting Models to Case–Control Edata
• Sample Selection Bias Correction Theory
• Separate sample logistic discrimination

Cited in

(25)

• Optimal subsampling for large-scale quantile regression
• Surprise sampling: improving and extending the local case-control sampling
• A two-stage optimal subsampling estimation for missing data problems with large-scale data
• Information-based optimal subdata selection for big data logistic regression
• Optimal subsampling for softmax regression
• Automated scalable Bayesian inference via Hilbert coresets
• Local case-control sampling: efficient subsampling in imbalanced data sets
• Optimal subsampling for multiplicative regression with massive data
• Model constraints independent optimal subsampling probabilities for softmax regression
• A review on design inspired subsampling for big data
• Approximating Partial Likelihood Estimators via Optimal Subsampling
• Deterministic subsampling for logistic regression with massive data
• Optimal Poisson subsampling for softmax regression
• A Subsampling Method for Regression Problems Based on Minimum Energy Criterion
• Optimal subsampling for estimation of dimension reduction directions
• Efficient subsampling for high-dimensional data
• Refitted cross-validation estimation for high-dimensional subsamples from low-dimension full data
• Optimal sample selection through uncertainty estimation and its application in deep learning
• Optimal Subsampling for Data Streams with Measurement Constrained Categorical Responses
• Subsampled one-step estimation for fast statistical inference
• DsubCox: a fast subsampling algorithm for Cox model with distributed and massive survival data
• Optimal subsampling for multinomial logistic models with big data
• A Subsampling Strategy for AIC-based Model Averaging with Generalized Linear Models
• Independence-Encouraging Subsampling for Nonparametric Additive Models
• Distributed information-based optimal sub-data selection algorithm for big data logistic regression

This page was built for publication: Local uncertainty sampling for large-scale multiclass logistic regression

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2196246)