Affine symmetries and neural network identifiability

DOI10.1016/J.AIM.2020.107485zbMATH Open1468.92016arXiv2006.11727OpenAlexW3100101337MaRDI QIDQ2214101FDOQ2214101

Helmut Bölcskei, Verner Vlačić

Publication date: 4 December 2020

Published in: Advances in Mathematics (Search for Journal in Brave)

Abstract: We address the following question of neural network identifiability: Suppose we are given a function $f:mathbb{R}^{m}omathbb{R}^n$ and a nonlinearity $ho$. Can we specify the architecture, weights, and biases of all feed-forward neural networks with respect to $ho$ giving rise to $f$? Existing literature on the subject suggests that the answer should be yes, provided we are only concerned with finding networks that satisfy certain "genericity conditions". Moreover, the identified networks are mutually related by symmetries of the nonlinearity. For instance, the $anh$ function is odd, and so flipping the signs of the incoming and outgoing weights of a neuron does not change the output map of the network. The results known hitherto, however, apply either to single-layer networks, or to networks satisfying specific structural assumptions (such as full connectivity), as well as to specific nonlinearities. In an effort to answer the identifiability question in greater generality, we consider arbitrary nonlinearities with potentially complicated affine symmetries, and we show that the symmetries can be used to find a rich set of networks giving rise to the same function $f$. The set obtained in this manner is, in fact, exhaustive (i.e., it contains all networks giving rise to $f$) unless there exists a network $mathcalA$ "with no internal symmetries" giving rise to the identically zero function. This result can thus be interpreted as an analog of the rank-nullity theorem for linear operators. We furthermore exhibit a class of "$anh$-type" nonlinearities (including the tanh function itself) for which such a network $mathcalA$ does not exist, thereby solving the identifiability question for these nonlinearities in full generality. Finally, we show that this class contains nonlinearities with arbitrarily complicated symmetries.

Full work available at URL: https://arxiv.org/abs/2006.11727

zbMATH Keywords

analytic continuationtanhaffine symmetriesneural network identifiabilityneural network parametrization

Mathematics Subject Classification ID

Neural networks for/in biological studies, artificial life and related topics (92B20) System identification (93B30)

Cites Work

• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• Group invariant scattering
• Optimal approximation of piecewise smooth functions using deep ReLU neural networks
• Optimal Approximation with Sparsely Connected Deep Neural Networks
• Reconstructing a neural net from its output
• A Mathematical Theory of Deep Convolutional Neural Networks for Feature Extraction

Cited In (5)

• Stable recovery of entangled weights: towards robust identification of deep neural networks from minimal samples
• Metric entropy limits on recurrent neural network learning of linear dynamical systems
• Gauge symmetry and neural networks
• Information theory and recovery algorithms for data fusion in Earth observation
• Identification in the presence of symmetry: oscillator networks

Uses Software

• ImageNet
• AlexNet

This page was built for publication: Affine symmetries and neural network identifiability

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2214101)