Hybrid tensor decomposition in neural network compression

DOI10.1016/J.NEUNET.2020.09.006zbMATH Open1475.68325DBLPjournals/nn/WuWZDL20arXiv2006.15938OpenAlexW3037399553WikidataQ99724362 ScholiaQ99724362MaRDI QIDQ2057771FDOQ2057771

Authors: Bijiao Wu, Dingheng Wang, Guangshe Zhao, Lei Deng, Guoqi Li 

Publication date: 7 December 2021

Published in: Neural Networks (Search for Journal in Brave)

Abstract: Deep neural networks (DNNs) have enabled impressive breakthroughs in various artificial intelligence (AI) applications recently due to its capability of learning high-level features from big data. However, the current demand of DNNs for computational resources especially the storage consumption is growing due to that the increasing sizes of models are being required for more and more complicated applications. To address this problem, several tensor decomposition methods including tensor-train (TT) and tensor-ring (TR) have been applied to compress DNNs and shown considerable compression effectiveness. In this work, we introduce the hierarchical Tucker (HT), a classical but rarely-used tensor decomposition method, to investigate its capability in neural network compression. We convert the weight matrices and convolutional kernels to both HT and TT formats for comparative study, since the latter is the most widely used decomposition method and the variant of HT. We further theoretically and experimentally discover that the HT format has better performance on compressing weight matrices, while the TT format is more suited for compressing convolutional kernels. Based on this phenomenon we propose a strategy of hybrid tensor decomposition by combining TT and HT together to compress convolutional and fully connected parts separately and attain better accuracy than only using the TT or HT format on convolutional neural networks (CNNs). Our work illuminates the prospects of hybrid tensor decomposition for neural network compression.

Full work available at URL: https://arxiv.org/abs/2006.15938

Recommendations

• CMD: controllable matrix decomposition with global optimization for deep neural network compression
• Towards compact neural networks via end-to-end training: a Bayesian tensor approach with automatic rank determination
• Survey of deep neural network model compression
• Tensor neural network models for tensor singular value decompositions
• Tensor Networks for Dimensionality Reduction and Large-scale Optimization: Part 2 Applications and Future Perspectives

zbMATH Keywords

balanced structurehierarchical Tuckertensor-trainhybrid tensor decompositionneural network compression

Mathematics Subject Classification ID

Artificial neural networks and deep learning (68T07) Factorization of matrices (15A23) Multilinear algebra, tensor calculus (15A69)

Cites Work

• Analysis of individual differences in multidimensional scaling via an \(n\)-way generalization of ``Eckart-Young decomposition
• Tensor-train decomposition
• Hierarchical Singular Value Decomposition of Tensors
• Optimization problems in contracted tensor networks
• A new scheme for the tensor representation
• Algorithm 941: \texttt{htucker} -- a Matlab toolbox for tensors in hierarchical Tucker format
• Decompositions of a Higher-Order Tensor in Block Terms—Part II: Definitions and Uniqueness
• An introduction to hierarchical (\(\mathcal H\)-) rank and TT-rank of tensors with examples
• Preconditioned low-rank methods for high-dimensional elliptic PDE eigenvalue problems
• Regularized computation of approximate pseudoinverse of large matrices using low-rank tensor train decompositions

Cited In (6)

• Towards compact neural networks via end-to-end training: a Bayesian tensor approach with automatic rank determination
• Survey of deep neural network model compression
• A zeroing neural dynamics based acceleration optimization approach for optimizers in deep neural networks
• Title not available (Why is that?)
• CMD: controllable matrix decomposition with global optimization for deep neural network compression
• Online subspace learning and imputation by tensor-ring decomposition

Uses Software

• htucker
• LOBPCG
• MNIST
• CIFAR
• ImageNet
• SqueezeDet
• AlexNet

This page was built for publication: Hybrid tensor decomposition in neural network compression

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2057771)