On the number of regions of piecewise linear neural networks
In the last decade the use of neural networks has had enormous success in many scientific disciplines. In particular in the application to train deep parametric models, applications range from computer vision, signal processing, language processing, neuroscience, learning in various forms and many more applications. For example, many popular deep models belong to the family of feedforward neural networks (NNs), for which the input-output mapping takes the form \(x\to \sigma_Lf_{\theta_L}\sigma_{L-1}f_{\theta_{L-1}} \cdots \sigma_2f_{\theta_2} \sigma_1f_{\theta_1}(x)\). Here, \(L\) is the number of layers of the NN (referred to as the depth of the NN), \(f_{\theta_k}:\mathbb R^{d_k}\to \mathbb R^{d_{k\to 1}}\) is an affine function parameterized by \(\theta_k\) and \(\sigma_k\) is a non-affine activation function. It is known that many feedforward neural networks (NNs) generate continuous and piecewise-linear (CPWL) mappings and partition the input domain into so-called linear regions on which the mapping is affine. The number of these regions induces a metric to characterize the expressiveness of CPWL NNs. Unfortunately, in practice the exact number of these regions is difficult to find and bounds have been proposed for specific architectures, including for ReLU and Maxout NNs. In this work, the authors generalize these bounds to NNs with arbitrary and possibly multivariate CPWL activation functions. They provide upper and lower bounds on the maximal number of linear regions of a CPWL NN given its depth, width, and the number of linear regions of its activation functions. Their results rely on the combinatorial structure of convex partitions and confirm the distinctive role of depth which, on its own, is able to exponentially increase the number of regions. The authors then introduce a stochastic type framework to estimate the average number of linear regions produced by a CPWL NN. Under various and reasonable assumptions, the expected density of linear regions along any 1D path is bounded by the product of depth, width, and a measure of activation complexity (up to a scaling factor). This yields the same role to the following three sources of expressiveness: no exponential growth with depth is seen anymore.
- Sharp Bounds for the Number of Regions of Maxout Networks and Vertices of Minkowski Sums
- On decision regions of narrow deep neural networks
- Lower and upper bounds for numbers of linear regions of graph convolutional networks
- scientific article; zbMATH DE number 7064043
- The capacity of feedforward neural networks
- A Framework for the Construction of Upper Bounds on the Number of Affine Linear Regions of ReLU Feed-Forward Neural Networks
- A representation method for PWL functions oriented to parallel processing
- A representer theorem for deep neural networks
- Approximation of Lipschitz Functions Using Deep Spline Neural Networks
- Deep vs. shallow networks: an approximation theory perspective
- Facing up to arrangements: face-count formulas for partitions of space by hyperplanes
- Generalization of Hinging Hyperplanes
- Learning deep architectures for AI
- Region configurations for realizability of lattice piecewise-linear models.
- Sharp Bounds for the Number of Regions of Maxout Networks and Vertices of Minkowski Sums
- Spaces of convex \(n\)-partitions
- Stable parameterization of continuous and piecewise-linear functions
- Sharp Bounds for the Number of Regions of Maxout Networks and Vertices of Minkowski Sums
- Lower and upper bounds for numbers of linear regions of graph convolutional networks
- Convergence of a multilayer perceptron to histogram-based Bayesian regression
- Spectral complexity of deep neural networks
- Convexity in ReLU neural networks: beyond ICNNs?
- Data-driven regularization with weak convexity for robust image reconstruction
This page was built for publication: On the number of regions of piecewise linear neural networks
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6145187)
