Sketching for large-scale learning of mixture models
From MaRDI portal
Abstract: Learning parameters from voluminous data can be prohibitive in terms of memory and computational requirements. We propose a "compressive learning" framework where we estimate model parameters from a sketch of the training data. This sketch is a collection of generalized moments of the underlying probability distribution of the data. It can be computed in a single pass on the training set, and is easily computable on streams or distributed datasets. The proposed framework shares similarities with compressive sensing, which aims at drastically reducing the dimension of high-dimensional signals while preserving the ability to reconstruct them. To perform the estimation task, we derive an iterative algorithm analogous to sparse reconstruction algorithms in the context of linear inverse problems. We exemplify our framework with the compressive estimation of a Gaussian Mixture Model (GMM), providing heuristics on the choice of the sketching procedure and theoretical guarantees of reconstruction. We experimentally show on synthetic data that the proposed algorithm yields results comparable to the classical Expectation-Maximization (EM) technique while requiring significantly less memory and fewer computations when the number of database elements is large. We further demonstrate the potential of the approach on real large-scale data (over 10 8 training samples) for the task of model-based speaker verification. Finally, we draw some connections between the proposed framework and approximate Hilbert space embedding of probability distributions using random features. We show that the proposed sketching operator can be seen as an innovative method to design translation-invariant kernels adapted to the analysis of GMMs. We also use this theoretical framework to derive information preservation guarantees, in the spirit of infinite-dimensional compressive sensing.
Recommendations
- Compressive Gaussian mixture estimation
- Compressive statistical learning with random feature moments
- Statistical learning guarantees for compressive clustering and compressive mixture modeling
- Compressive learning for patch-based image denoising
- Sketching meets random projection in the dual: a provable recovery algorithm for big and high-dimensional data
Cited in
(14)- Sketched learning for image denoising
- Estimation of off-the grid sparse spikes with over-parametrized projected gradient descent: theory and application
- Matrix inference and estimation in multi-layer models*
- Batch-less stochastic gradient descent for compressive learning of deep regularization for image denoising
- Joint Gaussian dictionary learning and tomographic reconstruction
- Compressive Gaussian mixture estimation
- Training Gaussian mixture models at scale via coresets
- Compressive statistical learning with random feature moments
- Statistical learning guarantees for compressive clustering and compressive mixture modeling
- Localization of point scatterers via sparse optimization on measures
- scientific article; zbMATH DE number 7255176 (Why is no real title available?)
- Compressive learning for patch-based image denoising
- The basins of attraction of the global minimizers of the non-convex sparse spike estimation problem
- Applied harmonic analysis and data processing. Abstracts from the workshop held March 25--31, 2018
This page was built for publication: Sketching for large-scale learning of mixture models
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5242851)
