Interpretable global minima of deep ReLU neural networks on sequentially separable data
This interesting paper studies interpretable global minima of deep ReLU neural networks on sequentially separable data. More specifically, the paper gives an interpretation for the action of the layers of a neural network, and explicitly constructs global minima for a classification task with well distributed data, in a way that is independent of the number of training samples. The characterization is achieved via a map encoding the action layer given by a weight matrix say \(W\), a bias vector say \(b\), and activation function say \(\sigma\) in the form of: \((W*)(\sigma(Wx+b)-b)\) with \(W*\) the generalized inverse of \(W\).The configurations for the training data considered are sufficiently small and are well separated clusters corresponding to each class. Morever, equivalence classes are sequentially linearly separable. For \(Q\) classes of data in \(\mathbb R^n\), global minimizers can be described with \(Q(n + 2)\) parameters.\N\NThe paper is well written with a good set of references.
- Generalized Inverses of Linear Transformations
- scientific article; zbMATH DE number 7064043 (Why is no real title available?)
- Neural collapse with unconstrained features
- On non-approximability of zero loss global \(\mathcal{L}^2\) minimizers by gradient descent in deep learning
- Prevalence of neural collapse during the terminal phase of deep learning training
This page was built for publication: Interpretable global minima of deep ReLU neural networks on sequentially separable data
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6887366)
