OAMIP: optimizing ANN architectures using mixed-integer programming
From MaRDI portal
Abstract: We introduce a mixed integer program (MIP) for assigning importance scores to each neuron in deep neural network architectures which is guided by the impact of their simultaneous pruning on the main learning task of the network. By carefully devising the objective function of the MIP, we drive the solver to minimize the number of critical neurons (i.e., with high importance score) that need to be kept for maintaining the overall accuracy of the trained neural network. Further, the proposed formulation generalizes the recently considered lottery ticket optimization by identifying multiple "lucky" sub-networks resulting in optimized architecture that not only performs well on a single dataset, but also generalizes across multiple ones upon retraining of network weights. Finally, we present a scalable implementation of our method by decoupling the importance scores across layers using auxiliary networks. We demonstrate the ability of our formulation to prune neural networks with marginal loss in accuracy and generalizability on popular datasets and architectures.
Cites work
- CVXPY: a Python-embedded modeling language for convex optimization
- Deep learning
- Deep neural networks and mixed integer linear optimization
- Efficient Training of Convolutional Deep Belief Networks in the Frequency Domain for Application to High-Resolution 2D and 3D Images
- Lossless compression of deep neural networks
Cited in
(2)
This page was built for publication: OAMIP: optimizing ANN architectures using mixed-integer programming
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6057263)
