Continuous limits of residual neural networks in case of large input data
From MaRDI portal
Abstract: Residual deep neural networks (ResNets) are mathematically described as interacting particle systems. In the case of infinitely many layers the ResNet leads to a system of coupled system of ordinary differential equations known as neural differential equations. For large scale input data we derive a mean--field limit and show well--posedness of the resulting description. Further, we analyze the existence of solutions to the training process by using both a controllability and an optimal control point of view. Numerical investigations based on the solution of a formal optimality system illustrate the theoretical findings.
Recommendations
- Deep limits of residual neural networks
- Stronger convergence results for deep residual networks: network width scales linearly with training data size
- Neural ODEs as the deep limit of ResNets with constant weights
- Continuity of approximation by neural networks in \(L_p\) spaces
- Deep stable neural networks: large-width asymptotics and convergence rates
- Rademacher complexity and the generalization error of residual networks
- An Integral Upper Bound for Neural Network Approximation
- Approximation capabilities of neural networks on unbounded domains
- Neural networks for optimal approximation of continuous functions in \(\mathbb R^d\)
Cites work
- A machine learning framework for data driven acceleration of computations of differential equations
- A mean field view of the landscape of two-layer neural networks
- A strategic learning algorithm for state-based games
- An artificial neural network as a troubled-cell indicator
- Constraint-aware neural networks for Riemann problems
- Control and nonlinearity.
- Controllability and observability of partial differential equations: some results and open problems
- Controlling oscillations in high-order discontinuous Galerkin schemes using artificial viscosity tuned by neural networks
- CWENO: uniformly accurate reconstructions for balance laws
- Deep learning observables in computational fluid dynamics
- Detecting troubled-cells on two-dimensional unstructured grids using a neural network
- Efficient implementation of weighted ENO schemes
- Ensemble Kalman inversion: a derivative-free technique for machine learning tasks
- Gradient flows in metric spaces and in the space of probability measures
- Learning in Modal Space: Solving Time-Dependent Stochastic PDEs Using Physics-Informed Neural Networks
- Mean field analysis of neural networks: a central limit theorem
- Mean field analysis of neural networks: a law of large numbers
- Mean-field and kinetic descriptions of neural differential equations
- Mean-field limit of a spatially-extended Fitzhugh-Nagumo neural network
- Multiscale modeling of pedestrian dynamics
- Numerical Optimization
- On the rate of convergence in Wasserstein distance of the empirical measure
- Optimal Transport
- Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations
- Replica-mean-field limits for intensity-based neural networks
- Simple bounds for the convergence of empirical and occupation measures in 1-Wasserstein distance
- Value Iteration Architecture Based Deep Learning for Intelligent Routing Exploiting Heterogeneous Computing Platforms
Cited in
(17)- Forward stability of ResNet and its variants
- Residual networks as flows of diffeomorphisms
- Selection dynamics for deep neural networks
- Interpolation and approximation via momentum ResNets and neural ODEs
- Analytic continuation of noisy data using Adams Bashforth residual neural network
- Deep limits of residual neural networks
- Approximation properties of residual neural networks for Kolmogorov PDEs
- Neural ODEs as the deep limit of ResNets with constant weights
- Stochastic training of residual networks in deep learning
- State-space representations of deep neural networks
- Doubly infinite residual neural networks: a diffusion process approach
- Stability of Deep Neural Networks via Discrete Rough Paths
- Convergence analysis of deep residual networks
- Mean-field limits of trained weights in deep learning: a dynamical systems perspective
- Variational formulations of ODE-Net as a mean-field optimal control problem and existence results
- Mini-workshop: High-dimensional control problems and mean-field equations with applications in machine learning. Abstracts from the mini-workshop held December 8--13, 2024
- Controllability of continuous networks and a kernel-based learning approximation
This page was built for publication: Continuous limits of residual neural networks in case of large input data
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6098879)
