On minimal representations of shallow ReLU networks

DOI10.1016/J.NEUNET.2022.01.006MaRDI QIDQ6072445zbMATH OpenOpenAlexFDO

Authors Steffen Dereich, Sebastian Kassing

Publication date 13 October 2023

Published in Neural Networks (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2108.05643

zbMATH Keywords

minimal representationsneural networksReLU activationshallow networks

Mathematics Subject Classification ID

Artificial intelligence (68Txx)

Abstract: The realization function of a shallow ReLU network is a continuous and piecewise affine function $f:mathbb{R}^{d}omathbbR$, where the domain $mathbb{R}^d$ is partitioned by a set of $n$ hyperplanes into cells on which $f$ is affine. We show that the minimal representation for $f$ uses either $n$, $n+1$ or $n+2$ neurons and we characterize each of the three cases. In the particular case, where the input layer is one-dimensional, minimal representations always use at most $n+1$ neurons but in all higher dimensional settings there are functions for which $n+2$ neurons are needed. Then we show that the set of minimal networks representing $f$ forms a $C^{i}nfty$-submanifold $M$ and we derive the dimension and the number of connected components of $M$. Additionally, we give a criterion for the hyperplanes that guarantees that all continuous, piecewise affine functions are realization functions of appropriate ReLU networks.

Recommendations

• Towards Lower Bounds on the Depth of ReLU Neural Networks
• Rates of approximation by ReLU shallow neural networks
• Deep vs. shallow networks: an approximation theory perspective
• On the complexity of loading shallow neural networks
• A representer theorem for deep neural networks
• ReLU deep neural networks from the hierarchical basis perspective
• Characterization of the variation spaces corresponding to shallow neural networks
• scientific article; zbMATH DE number 4211542
• Approximation properties of deep ReLU CNNs
• Deep ReLU neural networks in high-dimensional approximation

Cites work

• Approximation by superpositions of a sigmoidal function
• Convergence rates for the stochastic gradient descent method for non-convex objective functions
• Multilayer feedforward networks are universal approximators

Cited in

(8)

• ReLU deep neural networks from the hierarchical basis perspective
• An embedding of ReLU networks and an analysis of their identifiability
• A linear algebra approach to minimal convolutional encoders
• On Transversality of Bent Hyperplane Arrangements and the Topological Expressiveness of ReLU Neural Networks
• Towards Lower Bounds on the Depth of ReLU Neural Networks
• On the existence of minimizers in shallow residual ReLU neural network optimization landscapes
• Stochastic modified flows for Riemannian stochastic gradient descent
• On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networks

This page was built for publication: On minimal representations of shallow ReLU networks

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6072445)