Deep Neural Networks Pruning via the Structured Perspective Regularization

DOI10.1137/22M1542313zbMATH Open1530.68226arXiv2206.14056OpenAlexW4389339766MaRDI QIDQ6148352FDOQ6148352

Authors: Matteo Cacciola, Antonio Frangioni, Andrea Lodi 

Publication date: 11 January 2024

Published in: SIAM Journal on Mathematics of Data Science (Search for Journal in Brave)

Abstract: In Machine Learning, Artificial Neural Networks (ANNs) are a very powerful tool, broadly used in many applications. Often, the selected (deep) architectures include many layers, and therefore a large amount of parameters, which makes training, storage and inference expensive. This motivated a stream of research about compressing the original networks into smaller ones without excessively sacrificing performances. Among the many proposed compression approaches, one of the most popular is emph{pruning}, whereby entire elements of the ANN (links, nodes, channels, ldots) and the corresponding weights are deleted. Since the nature of the problem is inherently combinatorial (what elements to prune and what not), we propose a new pruning method based on Operational Research tools. We start from a natural Mixed-Integer-Programming model for the problem, and we use the Perspective Reformulation technique to strengthen its continuous relaxation. Projecting away the indicator variables from this reformulation yields a new regularization term, which we call the Structured Perspective Regularization, that leads to structured pruning of the initial architecture. We test our method on some ResNet architectures applied to CIFAR-10, CIFAR-100 and ImageNet datasets, obtaining competitive performances w.r.t.~the state of the art for structured pruning.

Full work available at URL: https://arxiv.org/abs/2206.14056

Recommendations

• Fast convex pruning of deep neural networks
• Sensitivity-informed provable pruning of neural networks
• Efficient and sparse neural networks by pruning weights in a multiobjective learning approach
• Pruning during training by network efficacy modeling

zbMATH Keywords

optimizationartificial neural networkspruning

Mathematics Subject Classification ID

Convex programming (90C25) Artificial neural networks and deep learning (68T07) Applications of mathematical programming (90C90) Integer programming (90C10)

Cites Work

• Regularization and Variable Selection Via the Elastic Net
• Robust Estimation of a Location Parameter
• Perspective cuts for a class of convex 0-1 mixed integer programs
• Approximated perspective relaxations: a project and lift approach
• Projected perspective reformulations with applications in design problems
• On mathematical programming with indicator constraints
• Title not available (Why is that?)
• Fast convex pruning of deep neural networks
• Lossless compression of deep neural networks
• Deep neural networks and mixed integer linear optimization
• The adaptive BerHu penalty in robust regression
• Sensitivity-informed provable pruning of neural networks
• OAMIP: optimizing ANN architectures using mixed-integer programming

This page was built for publication: Deep Neural Networks Pruning via the Structured Perspective Regularization

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6148352)