Hybrid tensor decomposition in neural network compression
From MaRDI portal
Publication:2057771
Abstract: Deep neural networks (DNNs) have enabled impressive breakthroughs in various artificial intelligence (AI) applications recently due to its capability of learning high-level features from big data. However, the current demand of DNNs for computational resources especially the storage consumption is growing due to that the increasing sizes of models are being required for more and more complicated applications. To address this problem, several tensor decomposition methods including tensor-train (TT) and tensor-ring (TR) have been applied to compress DNNs and shown considerable compression effectiveness. In this work, we introduce the hierarchical Tucker (HT), a classical but rarely-used tensor decomposition method, to investigate its capability in neural network compression. We convert the weight matrices and convolutional kernels to both HT and TT formats for comparative study, since the latter is the most widely used decomposition method and the variant of HT. We further theoretically and experimentally discover that the HT format has better performance on compressing weight matrices, while the TT format is more suited for compressing convolutional kernels. Based on this phenomenon we propose a strategy of hybrid tensor decomposition by combining TT and HT together to compress convolutional and fully connected parts separately and attain better accuracy than only using the TT or HT format on convolutional neural networks (CNNs). Our work illuminates the prospects of hybrid tensor decomposition for neural network compression.
Recommendations
- CMD: controllable matrix decomposition with global optimization for deep neural network compression
- Towards compact neural networks via end-to-end training: a Bayesian tensor approach with automatic rank determination
- Survey of deep neural network model compression
- Tensor neural network models for tensor singular value decompositions
- Tensor Networks for Dimensionality Reduction and Large-scale Optimization: Part 2 Applications and Future Perspectives
Cites work
- A new scheme for the tensor representation
- Algorithm 941: \texttt{htucker} -- a Matlab toolbox for tensors in hierarchical Tucker format
- An introduction to hierarchical (\(\mathcal H\)-) rank and TT-rank of tensors with examples
- Analysis of individual differences in multidimensional scaling via an \(n\)-way generalization of ``Eckart-Young decomposition
- Decompositions of a Higher-Order Tensor in Block Terms—Part II: Definitions and Uniqueness
- Hierarchical Singular Value Decomposition of Tensors
- Optimization problems in contracted tensor networks
- Preconditioned low-rank methods for high-dimensional elliptic PDE eigenvalue problems
- Regularized computation of approximate pseudoinverse of large matrices using low-rank tensor train decompositions
- Tensor-train decomposition
Cited in
(6)- Towards compact neural networks via end-to-end training: a Bayesian tensor approach with automatic rank determination
- Survey of deep neural network model compression
- A zeroing neural dynamics based acceleration optimization approach for optimizers in deep neural networks
- scientific article; zbMATH DE number 5150445 (Why is no real title available?)
- CMD: controllable matrix decomposition with global optimization for deep neural network compression
- Online subspace learning and imputation by tensor-ring decomposition
This page was built for publication: Hybrid tensor decomposition in neural network compression
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2057771)
