Consistent Sparse Deep Learning: Theory and Computation
From MaRDI portal
Abstract: Deep learning has been the engine powering many successes of data science. However, the deep neural network (DNN), as the basic model of deep learning, is often excessively over-parameterized, causing many difficulties in training, prediction and interpretation. We propose a frequentist-like method for learning sparse DNNs and justify its consistency under the Bayesian framework: the proposed method could learn a sparse DNN with at most connections and nice theoretical guarantees such as posterior consistency, variable selection consistency and asymptotically optimal generalization bounds. In particular, we establish posterior consistency for the sparse DNN with a mixture Gaussian prior, show that the structure of the sparse DNN can be consistently determined using a Laplace approximation-based marginal posterior inclusion probability approach, and use Bayesian evidence to elicit sparse DNNs learned by an optimization method such as stochastic gradient descent in multiple runs with different initializations. The proposed method is computationally more efficient than standard Bayesian methods for large-scale sparse DNNs. The numerical results indicate that the proposed method can perform very well for large-scale network compression and high-dimensional nonlinear variable selection, both advancing interpretable machine learning.
Recommendations
- Learning sparse deep neural networks with a spike-and-slab prior
- On the minimax optimality and superiority of deep neural network learning over sparse parameter spaces
- Deep Learning as Sparsity-Enforcing Algorithms
- Sparse Bayesian deep learning for dynamic system identification
- Sparse deep neural networks using \(L_{1,\infty}\)-weight normalization
Cites work
- An introduction to variational methods for graphical models
- Bayesian estimation of sparse signals with a continuous spike-and-slab prior
- Bayesian neural networks for selection of drug sensitive genes
- Bayesian Subset Modeling for High-Dimensional Generalized Linear Models
- Bayesian variable selection for high dimensional generalized linear models: convergence rates of the fitted densities
- Convergence rates of posterior distributions.
- Deep double descent: where bigger models and more data hurt*
- Entropy-SGD: biasing gradient descent into wide valleys
- Error bounds for approximations with deep ReLU networks
- Evidence Evaluation for Bayesian Neural Networks Using Contour Monte Carlo
- scientific article; zbMATH DE number 6378127 (Why is no real title available?)
- scientific article; zbMATH DE number 4215168 (Why is no real title available?)
- scientific article; zbMATH DE number 1034042 (Why is no real title available?)
- Model selection in Bayesian neural networks via horseshoe priors
- Nearly optimal Bayesian shrinkage for high-dimensional regression
- Nonparametric regression using deep neural networks with ReLU activation function
- On deep learning as a remedy for the curse of dimensionality in nonparametric regression
- Optimal approximation of piecewise smooth functions using deep ReLU neural networks
- Optimal approximation with sparsely connected deep neural networks
- Some PAC-Bayesian theorems
- Sparse graphical models for exploring gene expression data
- Spike and slab variable selection: frequentist and Bayesian strategies
- Transformed \(\ell_1\) regularization for learning sparse deep neural networks
- Weak Convergence Rates of Population Versus Single-Chain Stochastic Approximation MCMC Algorithms
Cited in
(28)- Deep learning: a Bayesian perspective
- SSN: learning sparse switchable normalization via SparsestMax
- A phase transition for finding needles in nonlinear haystacks with LASSO artificial neural networks
- Sparse Bayesian deep learning for dynamic system identification
- Transformed \(\ell_1\) regularization for learning sparse deep neural networks
- On the minimax optimality and superiority of deep neural network learning over sparse parameter spaces
- Inference, learning and attention mechanisms that exploit and preserve sparsity in CNNs
- Learning sparse deep neural networks with a spike-and-slab prior
- Sparse kernel deep stacking networks
- Dynamic sparse method for deep learning execution
- Working Locally Thinking Globally: Theoretical Guarantees for Convolutional Sparse Coding
- Nonconvex Sparse Regularization for Deep Neural Networks and Its Optimality
- scientific article; zbMATH DE number 7626756 (Why is no real title available?)
- Sparse deep neural networks using \(L_{1,\infty}\)-weight normalization
- Learning with Structured Sparsity
- Deep Learning as Sparsity-Enforcing Algorithms
- Minimax optimality of deep neural networks on dependent data via PAC-Bayes bounds
- Bayesian scalar-on-image regression with a spatially varying single-layer neural network prior
- Extended fiducial inference: toward an automated process of statistical inference
- Posterior concentrations of fully-connected Bayesian neural networks with general priors on the weights
- Extended fiducial inference for individual treatment effects via deep neural networks
- Bayesian autoencoders for data-driven discovery of coordinates, governing equations and fundamental constants
- A new paradigm for generative adversarial networks based on randomized decision rules
- Word-Level Maximum Mean Discrepancy Regularization for Word Embedding
- Variational Bayes neural networks for high-dimensional non parametric regression: Minimax optimality and adaptivity
- Deep P-Spline: Theory, Fast Tuning, and Application
- DNN-based semiparametric AFT model for integrating genomic and pathological imaging data in cancer prognosis
- Supervised Predictive Modeling of High-Dimensional Data with Group ℓ 0 -Norm Constrained Neural Networks
This page was built for publication: Consistent Sparse Deep Learning: Theory and Computation
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6110715)
