Dynamical Systems–Based Neural Networks

DOI10.1137/22M1527337arXiv2210.02373MaRDI QIDQ6181900FDOQ6181900

Elena Celledoni, Carola-Bibiane Schönlieb, Davide Murari, Ferdia Sherry, Brynjulf Owren

Publication date: 20 December 2023

Published in: SIAM Journal on Scientific Computing (Search for Journal in Brave)

Abstract: Neural networks have gained much interest because of their effectiveness in many applications. However, their mathematical properties are generally not well understood. If there is some underlying geometric structure inherent to the data or to the function to approximate, it is often desirable to take this into account in the design of the neural network. In this work, we start with a non-autonomous ODE and build neural networks using a suitable, structure-preserving, numerical time-discretisation. The structure of the neural network is then inferred from the properties of the ODE vector field. Besides injecting more structure into the network architectures, this modelling procedure allows a better theoretical understanding of their behaviour. We present two universal approximation results and demonstrate how to impose some particular properties on the neural networks. A particular focus is on 1-Lipschitz architectures including layers that are not 1-Lipschitz. These networks are expressive and robust against adversarial attacks, as shown for the CIFAR-10 dataset.

Full work available at URL: https://arxiv.org/abs/2210.02373

zbMATH Keywords

dynamical systemsneural networksuniversal approximation theoremLipschitz networksstructure-preserving deep learning

Mathematics Subject Classification ID

Artificial neural networks and deep learning (68T07) Multistep, Runge-Kutta and extrapolation methods for ordinary differential equations (65L06) Discretization methods and integrators (symplectic, variational, geometric, etc.) for dynamical systems (37M15) Numerical methods for initial value problems involving ordinary differential equations (65L05)

Cites Work

• Switching in systems and control
• Geometric Numerical Integration
• Basic problems in stability and design of switched systems
• Title not available (Why is that?)
• Simulating Hamiltonian Dynamics
• Splitting methods
• Title not available (Why is that?)
• \(L^p\) approximation of maps by diffeomorphisms
• A high-order conservative Patankar-type discretisation for stiff systems of production--destruction equations
• Positivity-preserving methods for ordinary differential equations
• Non-local decomposition of vector fields
• Structure-preserving deep learning
• Reducibility and contractivity of Runge-Kutta methods revisited
• Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations
• Deep neural networks motivated by partial differential equations
• Solving inverse problems using data-driven models
• A proposal on machine learning via dynamical systems
• Order theory for discrete gradient methods
• Deep limits of residual neural networks
• Convolutional proximal neural networks and plug-and-play algorithms
• Sympnets: intrinsic structure-preserving symplectic networks for identifying Hamiltonian systems
• Deep learning via dynamical systems: an approximation perspective
• Learning Hamiltonians of constrained mechanical systems
• Deep Hamiltonian networks based on symplectic integrators
• Locally-symplectic neural networks for learning volume-preserving dynamics
• The modern mathematics of deep learning
• Control on the manifolds of mappings with a view to the deep learning
• PDE-based group equivariant convolutional neural networks
• Hamiltonian Deep Neural Networks Guaranteeing Nonvanishing Gradients by Design
• Neural ODE Control for Classification, Approximation, and Transport

Cited In (9)

• Designing stable neural networks using convex analysis and ODEs
• Title not available (Why is that?)
• Dynamics and architecture for neural computation
• Inferring the dynamics of oscillatory systems using recurrent neural networks
• Phase portrait approximation using dynamic neural networks
• Self-organizing feedforward neural network and modeling of nonlinear dynamical system
• A recurrent neural network for modelling dynamical systems
• Constrained dynamics, stochastic numerical methods and the modeling of complex systems. Abstracts from the workshop held May 26--31, 2024
• Modeling of complex dynamic systems using differential neural networks with the incorporation of a priori knowledge

This page was built for publication: Dynamical Systems–Based Neural Networks

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6181900)