Deep neural networks, generic universal interpolation, and controlled ODEs
From MaRDI portal
(Redirected from Publication:5037577)
Abstract: A recent paradigm views deep neural networks as discretizations of certain controlled ordinary differential equations, sometimes called neural ordinary differential equations. We make use of this perspective to link expressiveness of deep networks to the notion of controllability of dynamical systems. Using this connection, we study an expressiveness property that we call universal interpolation, and show that it is generic in a certain sense. The universal interpolation property is slightly weaker than universal approximation, and disentangles supervised learning on finite training sets from generalization properties. We also show that universal interpolation holds for certain deep neural networks even if large numbers of parameters are left untrained, and are instead chosen randomly. This lends theoretical support to the observation that training with random initialization can be successful even when most parameters are largely unchanged through the training. Our results also explore what a minimal amount of trainable parameters in neural ordinary differential equations could be without giving up on expressiveness.
Recommendations
- Neural ODE Control for Classification, Approximation, and Transport
- Interpolation and approximation via momentum ResNets and neural ODEs
- Control on the manifolds of mappings with a view to the deep learning
- Deep learning as optimal control problems: models and numerical methods
- Expressivity of Deep Neural Networks
Cites work
- A proposal on machine learning via dynamical systems
- A tour of subriemannian geometries, their geodesics and applications
- Approximation by superpositions of a sigmoidal function
- Defining relations for classical Lie algebras of polynomial vector fields.
- scientific article; zbMATH DE number 3563392 (Why is no real title available?)
- scientific article; zbMATH DE number 1405266 (Why is no real title available?)
- Just interpolate: kernel ``ridgeless regression can generalize
- On Everywhere Dense Imbedding of Free Groups in Lie Groups
Cited in
(23)- Reinforcement learning and stochastic optimisation
- Interpolation and approximation via momentum ResNets and neural ODEs
- Control on the manifolds of mappings with a view to the deep learning
- Neural networks in Fréchet spaces
- Control of partial differential equations via physics-informed neural networks
- Stochastic integrals and Gelfand integration in Fréchet spaces
- Large Sample Mean-Field Stochastic Optimization
- Turnpike in optimal control of PDEs, ResNets, and beyond
- Universal regular conditional distributions via probabilistic transformers
- Neural ODE Control for Classification, Approximation, and Transport
- Pricing options on flow forwards by neural networks in a Hilbert space
- An optimal control framework for adaptive neural ODEs
- Statistical inference and neural network training based on stochastic difference model for air pollution and associated disease transmission
- Convolutional neural networks as 2-D systems
- A gradient flow on control space with rough initial condition
- Constructive approximate transport maps with normalizing flows
- Cluster-based classification with neural ODEs via control
- A mathematical perspective on transformers
- Fokker-Planck equations on homogeneous Lie groups and probabilistic counterparts
- Randomized signature methods in optimal portfolio selection
- Interpolation, approximation, and controllability of deep neural networks
- Generic controllability of equivariant systems and applications to particle systems and neural networks
- Controllability on landmark manifolds for shapes and neural ODEs
This page was built for publication: Deep neural networks, generic universal interpolation, and controlled ODEs
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5037577)
