More is Less: Inducing Sparsity via Overparameterization
From MaRDI portal
Abstract: In deep learning it is common to overparameterize neural networks, that is, to use more parameters than training samples. Quite surprisingly training the neural network via (stochastic) gradient descent leads to models that generalize very well, while classical statistics would suggest overfitting. In order to gain understanding of this implicit bias phenomenon we study the special case of sparse recovery (compressed sensing) which is of interest on its own. More precisely, in order to reconstruct a vector from underdetermined linear measurements, we introduce a corresponding overparameterized square loss functional, where the vector to be reconstructed is deeply factorized into several vectors. We show that, if there exists an exact solution, vanilla gradient flow for the overparameterized loss functional converges to a good approximation of the solution of minimal -norm. The latter is well-known to promote sparse solutions. As a by-product, our results significantly improve the sample complexity for compressed sensing via gradient flow/descent on overparameterized models derived in previous works. The theory accurately predicts the recovery rate in numerical experiments. Our proof relies on analyzing a certain Bregman divergence of the flow. This bypasses the obstacles caused by non-convexity and should be of independent interest.
Recommendations
- Deep learning: a statistical viewpoint
- Sparse optimization on measures with over-parameterized gradient descent
- Flat minima generalize for low-rank matrix recovery
- Gradient descent for deep matrix factorization: dynamics and implicit bias towards low rank
- Estimation of off-the grid sparse spikes with over-parametrized projected gradient descent: theory and application
Cited in
(8)- Optimization over a probability simplex
- Estimation of off-the grid sparse spikes with over-parametrized projected gradient descent: theory and application
- From NeurODEs to AutoencODEs: a mean-field control framework for width-varying neural networks
- Learning sparse features can lead to overfitting in neural networks
- Gradient descent for deep matrix factorization: dynamics and implicit bias towards low rank
- Flat minima generalize for low-rank matrix recovery
- Robust implicit regularization via weight normalization
- Kurdyka-Łojasiewicz exponent via Hadamard parametrization
This page was built for publication: More is Less: Inducing Sparsity via Overparameterization
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6386274)
