Transformed _1 regularization for learning sparse deep neural networks
From MaRDI portal
(Redirected from Publication:2185659)
Transformed \(\ell 1\) regularization for learning sparse deep neural networks
Transformed \(\ell 1\) regularization for learning sparse deep neural networks
Abstract: Deep neural networks (DNNs) have achieved extraordinary success in numerous areas. However, to attain this success, DNNs often carry a large number of weight parameters, leading to heavy costs of memory and computation resources. Overfitting is also likely to happen in such network when the training data are insufficient. These shortcomings severely hinder the application of DNNs in resource-constrained platforms. In fact, many network weights are known to be redundant and can be removed from the network without much loss of performance. To this end, we introduce a new non-convex integrated transformed regularizer to promote sparsity for DNNs, which removes both redundant connections and unnecessary neurons simultaneously. To be specific, we apply the transformed to the matrix space of network weights and utilize it to remove redundant connections. Besides, group sparsity is also employed as an auxiliary to remove unnecessary neurons. An efficient stochastic proximal gradient algorithm is presented to solve the new model at the same time. To the best of our knowledge, this is the first work to utilize a non-convex regularizer in sparse optimization based method to promote sparsity for DNNs. Experiments on several public datasets demonstrate the effectiveness of the proposed method.
Recommendations
- Nonconvex regularization for sparse neural networks
- Nonconvex Sparse Regularization for Deep Neural Networks and Its Optimality
- Sparse deep neural networks using \(L_{1,\infty}\)-weight normalization
- \(L_{1/2}\) regularization methods for weights sparsification of neural networks
- Learning Sparsifying Transforms
- Sparsifying Transform Learning With Efficient Optimal Updates and Convergence Guarantees
- Make _1 regularization effective in training sparse CNN
- Deep Learning as Sparsity-Enforcing Algorithms
- Neural network for a class of sparse optimization with \(L_0\)-regularization
- Consistent Sparse Deep Learning: Theory and Computation
Cites work
- scientific article; zbMATH DE number 6378127 (Why is no real title available?)
- A method for finding structured sparse solutions to nonnegative least squares problems with applications
- A unified approach to model selection and sparse recovery using regularized least squares
- Analysis of multi-stage convex relaxation for sparse regularization
- Click Prediction for Web Image Reranking Using Multimodal Sparse Coding
- Computing sparse representation in a highly coherent dictionary based on difference of L₁ and L₂
- Deep learning
- Deep learning: methods and applications
- Enhancing sparsity by reweighted \(\ell _{1}\) minimization
- For most large underdetermined systems of linear equations the minimal 𝓁1‐norm solution is also the sparsest solution
- Local Strong Homogeneity of a Regularized Estimator
- Minimization of \(\ell_{1-2}\) for compressed sensing
- Minimization of transformed L₁ penalty: theory, difference of convex function algorithm, and robust application in compressed sensing
- Minimization of transformed \(l_1\) penalty: closed form representation and iterative thresholding algorithms
- Model Selection and Estimation in Regression with Grouped Variables
- Multi-Modal Curriculum Learning for Semi-Supervised Image Classification
- Nearly unbiased variable selection under minimax concave penalty
- Sparse Approximate Solutions to Linear Systems
- SparseNet: coordinate descent with nonconvex penalties
- Transformed Schatten-1 iterative thresholding algorithms for low rank matrix completion
- Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties
Cited in
(18)- Sparse deep neural networks using \(L_{1,\infty}\)-weight normalization
- Fast convex pruning of deep neural networks
- Dynamic sparse method for deep learning execution
- Calibrating multi-dimensional complex ODE from noisy data via deep neural networks
- A phase transition for finding needles in nonlinear haystacks with LASSO artificial neural networks
- Nonconvex regularization for sparse neural networks
- A brain-inspired algorithm for training highly sparse neural networks
- On obtaining sparse semantic solutions for inverse problems, control, and neural network training
- Sparsity-control ternary weight networks
- \(L_{1/2}\) regularization methods for weights sparsification of neural networks
- Nonconvex Sparse Regularization for Deep Neural Networks and Its Optimality
- Training thinner and deeper neural networks: jumpstart regularization
- Learning Sparsifying Transforms
- CMD: controllable matrix decomposition with global optimization for deep neural network compression
- Make _1 regularization effective in training sparse CNN
- Consistent Sparse Deep Learning: Theory and Computation
- Learning sparse deep neural networks with a spike-and-slab prior
- GSDAR: a fast Newton algorithm for \(\ell_0\) regularized generalized linear models with statistical guarantee
This page was built for publication: Transformed \(\ell_1\) regularization for learning sparse deep neural networks
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2185659)
