Exploring the common principal subspace of deep features in neural networks

DOI10.1007/S10994-021-06076-6MaRDI QIDQ2673330zbMATH OpenOpenAlexWikidataFDO

Authors Hao-Ran Liu, Haoyi Xiong, Yaqing Wang, Haozhe An, Dejing Dou, Dongrui Wu

Publication date 9 June 2022

Published in Machine Learning (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2110.02863

zbMATH Keywords

feature learninginterpretability of deep learningsubspaces of deep features

Mathematics Subject Classification ID

Artificial neural networks and deep learning (68T07)

Abstract: We find that different Deep Neural Networks (DNNs) trained with the same dataset share a common principal subspace in latent spaces, no matter in which architectures (e.g., Convolutional Neural Networks (CNNs), Multi-Layer Preceptors (MLPs) and Autoencoders (AEs)) the DNNs were built or even whether labels have been used in training (e.g., supervised, unsupervised, and self-supervised learning). Specifically, we design a new metric $mathcalP$-vector to represent the principal subspace of deep features learned in a DNN, and propose to measure angles between the principal subspaces using $mathcalP$-vectors. Small angles (with cosine close to $1.0$) have been found in the comparisons between any two DNNs trained with different algorithms/architectures. Furthermore, during the training procedure from random scratch, the angle decrease from a larger one ($70^{c}irc-80^{c}irc$ usually) to the small one, which coincides the progress of feature space learning from scratch to convergence. Then, we carry out case studies to measure the angle between the $mathcalP$-vector and the principal subspace of training dataset, and connect such angle with generalization performance. Extensive experiments with practically-used Multi-Layer Perceptron (MLPs), AEs and CNNs for classification, image reconstruction, and self-supervised learning tasks on MNIST, CIFAR-10 and CIFAR-100 datasets have been done to support our claims with solid evidences. Interpretability of Deep Learning, Feature Learning, and Subspaces of Deep Features

Recommendations

• Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
• On principal subspace analysis
• Transport analysis of infinitely deep neural network
• Emergence of invariance and disentanglement in deep representations
• Linear feature transform and enhancement of classification on deep neural network

Cites work

• A mathematical theory of semantic development in deep neural networks
• Finding structure with randomness: probabilistic algorithms for constructing approximate matrix decompositions
• scientific article; zbMATH DE number 823069 (Why is no real title available?)
• Numerical Methods for Computing Angles Between Linear Subspaces
• On the geometry of similarity search: dimensionality curse and concentration of measure

Describes a project that uses

Uses Software

• AlexNet
• SimCLR
• MNIST
• CIFAR
• GANDissect
• RankAggreg
• ImageNet

This page was built for publication: Exploring the common principal subspace of deep features in neural networks

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2673330)