Geometry of polynomial neural networks
The article presents a comprehensive algebro-geometric study of polynomial neural networks (PNNs). These are feedforward networks with polynomial activation functions, specifically monomial maps \(x \mapsto x^r\). The primary objects of study are the \textit{neuromanifold} \(\mathcal{M}_{d,r}\), defined as the image of the network's parameter map, and the \textit{neurovariety} \(\mathcal{V}_{d,r}\), its Zariski closure. The authors aim to characterize the expressivity of these networks (via the dimension of \(\mathcal{V}_{d,r}\)) and the complexity of their optimization landscape.\N\NThe first part of the paper addresses the expressivity of shallow PNNs. The authors establish a correspondence between these networks and symmetric tensor decompositions. By analyzing architectures such as \((d_0, d_1, 1)\) and \((d_0, 1, d_2)\), they identify conditions under which the architecture is ``filling (i.e., the neurovariety coincides with the ambient space of polynomials). The authors relate the dimension of the neurovariety to the classical Alexander-Hirschowitz theorem and provide specific dimension bounds. A key geometric insight is the distinction between the neuromanifold and the neurovariety; the authors prove that for architectures like \(d=(2,2,2)\), the neuromanifold is strictly contained within the neurovariety, and they describe the algebraic boundary separating them.\N\NThe second part focuses on the optimization landscape, specifically empirical risk minimization with Euclidean loss. The authors introduce the \textit{learning degree}, defined as the number of critical points of the loss function on the regular locus of the neurovariety. They prove that the learning degree is bounded above by the generic Euclidean distance (ED) degree of the neurovariety.\N\NA significant methodological contribution is the computation of this bound for the specific architecture \(d=(2,2,k)\) with quadratic activation (\(r=2\)). The associated neurovariety is a determinantal variety of rank-deficient matrices, which is singular. To overcome the difficulties posed by these singularities, the authors employ Chern-Mather classes to compute the generic ED degree. They derive the explicit formula \(8k^2 - 12k + 3\) for the ED degree in this case.\N\NThe paper concludes with numerical experiments on the gradient descent trajectory, supporting the theoretical bounds on the learning degree. The work effectively bridges algebraic statistics and deep learning theory, offering both structural results on network geometry and quantitative bounds on training complexity.
- Chern classes and characteristic cycles of determinantal varieties
- Decomposition of quantics in sums of powers of linear forms
- Determinantal rings
- Gradient descent learns linear dynamical systems
- scientific article; zbMATH DE number 5968745 (Why is no real title available?)
- scientific article; zbMATH DE number 704831 (Why is no real title available?)
- scientific article; zbMATH DE number 773851 (Why is no real title available?)
- Metric algebraic geometry
- Multilayer feedforward networks are universal approximators
- On real typical ranks
- On the typical rank of real binary forms
- Polar Classes and Segre Classes on Singular Projective Varieties
- Real rank geometry of ternary forms
- Symmetric Tensors and Symmetric Tensor Rank
- The implicit bias of gradient descent on separable data
- Tropical quadrics through three points
This page was built for publication: Geometry of polynomial neural networks
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q7030093)
