Inference, learning and attention mechanisms that exploit and preserve sparsity in CNNs
From MaRDI portal
Publication:2193592
Abstract: While CNNs naturally lend themselves to densely sampled data, and sophisticated implementations are available, they lack the ability to efficiently process sparse data. In this work we introduce a suite of tools that exploit sparsity in both the feature maps and the filter weights, and thereby allow for significantly lower memory footprints and computation times than the conventional dense framework when processing data with a high degree of sparsity. Our scheme provides (i) an efficient GPU implementation of a convolution layer based on direct, sparse convolution; (ii) a filter step within the convolution layer, which we call attention, that prevents fill-in, i.e., the tendency of convolution to rapidly decrease sparsity, and guarantees an upper bound on the computational resources; and (iii) an adaptation of the back-propagation algorithm, which makes it possible to combine our approach with standard learning frameworks, while still exploiting sparsity in the data and the model.
Recommendations
- Convolutional neural networks analyzed via convolutional sparse coding
- Consistent Sparse Deep Learning: Theory and Computation
- Learning sparse deep neural networks with a spike-and-slab prior
- scientific article; zbMATH DE number 7626756
- Deep Learning as Sparsity-Enforcing Algorithms
- Working Locally Thinking Globally: Theoretical Guarantees for Convolutional Sparse Coding
- Multi-Layer Convolutional Sparse Modeling: Pursuit and Dictionary Learning
- Variations on the Convolutional Sparse Coding Model
Cites work
Cited in
(3)
Describes a project that uses
Uses Software
This page was built for publication: Inference, learning and attention mechanisms that exploit and preserve sparsity in CNNs
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2193592)
