Deep stable neural networks: large-width asymptotics and convergence rates

DOI10.3150/22-BEJ1553arXiv2108.02316MaRDI QIDQ6103259FDOQ6103259

Authors: Stefano Favaro, S. Fortini, Stefano Peluchetti 

Publication date: 2 June 2023

Published in: Bernoulli (Search for Journal in Brave)

Abstract: In modern deep learning, there is a recent and growing literature on the interplay between large-width asymptotic properties of deep Gaussian neural networks (NNs), i.e. deep NNs with Gaussian-distributed weights, and Gaussian stochastic processes (SPs). Such an interplay has proved to be critical in Bayesian inference under Gaussian SP priors, kernel regression for infinitely wide deep NNs trained via gradient descent, and information propagation within infinitely wide NNs. Motivated by empirical analyses that show the potential of replacing Gaussian distributions with Stable distributions for the NN's weights, in this paper we present a rigorous analysis of the large-width asymptotic behaviour of (fully connected) feed-forward deep Stable NNs, i.e. deep NNs with Stable-distributed weights. We show that as the width goes to infinity jointly over the NN's layers, i.e. the ``joint growth" setting, a rescaled deep Stable NN converges weakly to a Stable SP whose distribution is characterized recursively through the NN's layers. Because of the non-triangular structure of the NN, this is a non-standard asymptotic problem, to which we propose an inductive approach of independent interest. Then, we establish sup-norm convergence rates of the rescaled deep Stable NN to the Stable SP, under the ``joint growth" and a ``sequential growth" of the width over the NN's layers. Such a result provides the difference between the ``joint growth" and the ``sequential growth" settings, showing that the former leads to a slower rate than the latter, depending on the depth of the layer and the number of inputs of the NN. Our work extends some recent results on infinitely wide limits for deep Gaussian NNs to the more general deep Stable NNs, providing the first result on convergence rates in the ``joint growth" setting.

Full work available at URL: https://arxiv.org/abs/2108.02316

Recommendations

• -Stable convergence of heavy-/light-tailed infinitely wide neural networks
• Infinite-width limit of deep linear neural networks
• Random neural networks in the infinite width limit as Gaussian processes
• Mean Field Analysis of Deep Neural Networks

zbMATH Keywords

Bayesian inferencespectral measureGaussian stochastic processdeep neural networkexchangeable sequencedepth limitneural tangent kernelstable stochastic processinfinitely wide limitsup-norm convergence rate

Mathematics Subject Classification ID

Bayesian inference (62F15) Gaussian processes (60G15) Artificial neural networks and deep learning (68T07) Stable stochastic processes (60G52)

Cites Work

• Gaussian processes for machine learning.
• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• Approximation of multidimensional stable densities
• Merging of Opinions with Increasing Information
• Inequalities for the $r$th Absolute Moment of a Sum of Random Variables, $1 \leqq r \leqq 2$
• Metrics for multivariate stable distributions
• Bayesian learning for neural networks
• Central Limit Theorems for Interchangeable Processes
• Wide neural networks of any depth evolve as linear models under gradient descent ^*
• Bayesian neural network priors for edge-preserving inversion

Cited In (5)

• Continuous limits of residual neural networks in case of large input data
• Gaussian random field approximation via Stein's method with applications to wide random neural networks
• On the existence of stable equilibrium points for CNNs
• -Stable convergence of heavy-/light-tailed infinitely wide neural networks
• On the S-instability and degeneracy of discrete deep learning models

This page was built for publication: Deep stable neural networks: large-width asymptotics and convergence rates

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6103259)