Geometry and generalization: eigenvalues as predictors of where a network will fail to generalize

DOI10.3934/FODS.2022004MaRDI QIDQ2148963zbMATH OpenOpenAlexFDO

Authors Susama Agarwala, Ben K. Dees, Andrew Gearheart, Corey Lowman

Publication date 24 June 2022

Published in Foundations of Data Science (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2107.06386

zbMATH Keywords

data representation theorydifferential geometryeigenvalues of local Jacobiansneural networks

Mathematics Subject Classification ID

Differential geometric aspects of statistical manifolds and information geometry (53B12) Applications of differential geometry to data and computer science (53Z50) Theory of computing (68Q99)

Abstract: We study the deformation of the input space by a trained autoencoder via the Jacobians of the trained weight matrices. In doing so, we prove bounds for the mean squared errors for points in the input space, under assumptions regarding the orthogonality of the eigenvectors. We also show that the trace and the product of the eigenvalues of the Jacobian matrices is a good predictor of the MSE on test points. This is a dataset independent means of testing an autoencoder's ability to generalize on new input. Namely, no knowledge of the dataset on which the network was trained is needed, only the parameters of the trained model.

Recommendations

• Geometry of learning in multilayer perceptrons
• Geometry of learning: Visualizing the performance of neural network supervised training methods
• Geometrical aspects of discrimination by multilayer perceptrons
• High-dimensional dynamics of generalization error in neural networks
• scientific article; zbMATH DE number 2095982
• scientific article; zbMATH DE number 4211541
• A singular Riemannian geometry approach to deep neural networks. I: Theoretical foundations
• Hausdorff dimension, heavy tails, and generalization in neural networks*

Cites work

• An Introduction to Variational Autoencoders
• Matrix inequalities
• Nonlinear Dimensionality Reduction
• Overparameterized neural networks implement associative memory

Describes a project that uses

Uses Software

• UMAP
• GitHub

This page was built for publication: Geometry and generalization: eigenvalues as predictors of where a network will fail to generalize

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2148963)