Transformed _1 regularization for learning sparse deep neural networks
From MaRDI portal
Publication:2185659
DOI10.1016/J.NEUNET.2019.08.015zbMATH Open1434.68512DBLPjournals/nn/MaMNZ19arXiv1901.01021OpenAlexW2970738028WikidataQ93198988 ScholiaQ93198988MaRDI QIDQ2185659FDOQ2185659
Authors: Yanyan Li 
Publication date: 5 June 2020
Published in: Neural Networks (Search for Journal in Brave)
Abstract: Deep neural networks (DNNs) have achieved extraordinary success in numerous areas. However, to attain this success, DNNs often carry a large number of weight parameters, leading to heavy costs of memory and computation resources. Overfitting is also likely to happen in such network when the training data are insufficient. These shortcomings severely hinder the application of DNNs in resource-constrained platforms. In fact, many network weights are known to be redundant and can be removed from the network without much loss of performance. To this end, we introduce a new non-convex integrated transformed regularizer to promote sparsity for DNNs, which removes both redundant connections and unnecessary neurons simultaneously. To be specific, we apply the transformed to the matrix space of network weights and utilize it to remove redundant connections. Besides, group sparsity is also employed as an auxiliary to remove unnecessary neurons. An efficient stochastic proximal gradient algorithm is presented to solve the new model at the same time. To the best of our knowledge, this is the first work to utilize a non-convex regularizer in sparse optimization based method to promote sparsity for DNNs. Experiments on several public datasets demonstrate the effectiveness of the proposed method.
Full work available at URL: https://arxiv.org/abs/1901.01021
Recommendations
- Nonconvex regularization for sparse neural networks
- Nonconvex Sparse Regularization for Deep Neural Networks and Its Optimality
- Sparse deep neural networks using \(L_{1,\infty}\)-weight normalization
- \(L_{1/2}\) regularization methods for weights sparsification of neural networks
- Learning Sparsifying Transforms
- Sparsifying Transform Learning With Efficient Optimal Updates and Convergence Guarantees
- Make \(\ell_1\) regularization effective in training sparse CNN
- Deep Learning as Sparsity-Enforcing Algorithms
- Neural network for a class of sparse optimization with \(L_0\)-regularization
- Consistent Sparse Deep Learning: Theory and Computation
Cites Work
- Nearly unbiased variable selection under minimax concave penalty
- SparseNet: coordinate descent with nonconvex penalties
- A unified approach to model selection and sparse recovery using regularized least squares
- Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties
- Analysis of multi-stage convex relaxation for sparse regularization
- Title not available (Why is that?)
- Model Selection and Estimation in Regression with Grouped Variables
- Enhancing sparsity by reweighted \(\ell _{1}\) minimization
- Deep learning
- Local Strong Homogeneity of a Regularized Estimator
- For most large underdetermined systems of linear equations the minimal 𝓁1‐norm solution is also the sparsest solution
- Sparse Approximate Solutions to Linear Systems
- Computing sparse representation in a highly coherent dictionary based on difference of \(L_1\) and \(L_2\)
- Minimization of \(\ell_{1-2}\) for compressed sensing
- A method for finding structured sparse solutions to nonnegative least squares problems with applications
- Deep learning: methods and applications
- Click Prediction for Web Image Reranking Using Multimodal Sparse Coding
- Transformed Schatten-1 iterative thresholding algorithms for low rank matrix completion
- Minimization of transformed \(L_1\) penalty: theory, difference of convex function algorithm, and robust application in compressed sensing
- Minimization of transformed \(l_1\) penalty: closed form representation and iterative thresholding algorithms
- Multi-Modal Curriculum Learning for Semi-Supervised Image Classification
Cited In (18)
- Fast convex pruning of deep neural networks
- Calibrating multi-dimensional complex ODE from noisy data via deep neural networks
- Dynamic sparse method for deep learning execution
- A phase transition for finding needles in nonlinear haystacks with LASSO artificial neural networks
- Nonconvex regularization for sparse neural networks
- A brain-inspired algorithm for training highly sparse neural networks
- On obtaining sparse semantic solutions for inverse problems, control, and neural network training
- Sparsity-control ternary weight networks
- \(L_{1/2}\) regularization methods for weights sparsification of neural networks
- Nonconvex Sparse Regularization for Deep Neural Networks and Its Optimality
- Learning Sparsifying Transforms
- Training thinner and deeper neural networks: jumpstart regularization
- CMD: controllable matrix decomposition with global optimization for deep neural network compression
- Make \(\ell_1\) regularization effective in training sparse CNN
- Consistent Sparse Deep Learning: Theory and Computation
- Learning sparse deep neural networks with a spike-and-slab prior
- GSDAR: a fast Newton algorithm for \(\ell_0\) regularized generalized linear models with statistical guarantee
- Sparse deep neural networks using \(L_{1,\infty}\)-weight normalization
Uses Software
This page was built for publication: Transformed \(\ell_1\) regularization for learning sparse deep neural networks
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2185659)
