Deep neural networks, generic universal interpolation, and controlled ODEs
From MaRDI portal
Publication:5037577
Abstract: A recent paradigm views deep neural networks as discretizations of certain controlled ordinary differential equations, sometimes called neural ordinary differential equations. We make use of this perspective to link expressiveness of deep networks to the notion of controllability of dynamical systems. Using this connection, we study an expressiveness property that we call universal interpolation, and show that it is generic in a certain sense. The universal interpolation property is slightly weaker than universal approximation, and disentangles supervised learning on finite training sets from generalization properties. We also show that universal interpolation holds for certain deep neural networks even if large numbers of parameters are left untrained, and are instead chosen randomly. This lends theoretical support to the observation that training with random initialization can be successful even when most parameters are largely unchanged through the training. Our results also explore what a minimal amount of trainable parameters in neural ordinary differential equations could be without giving up on expressiveness.
Recommendations
- Neural ODE Control for Classification, Approximation, and Transport
- Interpolation and approximation via momentum ResNets and neural ODEs
- Control on the manifolds of mappings with a view to the deep learning
- Deep learning as optimal control problems: models and numerical methods
- Expressivity of Deep Neural Networks
Cites work
- scientific article; zbMATH DE number 3563392 (Why is no real title available?)
- scientific article; zbMATH DE number 1405266 (Why is no real title available?)
- A proposal on machine learning via dynamical systems
- A tour of subriemannian geometries, their geodesics and applications
- Approximation by superpositions of a sigmoidal function
- Defining relations for classical Lie algebras of polynomial vector fields.
- Just interpolate: kernel ``ridgeless regression can generalize
- On Everywhere Dense Imbedding of Free Groups in Lie Groups
Cited in
(13)- Statistical inference and neural network training based on stochastic difference model for air pollution and associated disease transmission
- Interpolation and approximation via momentum ResNets and neural ODEs
- Reinforcement learning and stochastic optimisation
- Pricing options on flow forwards by neural networks in a Hilbert space
- Neural ODE Control for Classification, Approximation, and Transport
- Neural networks in Fréchet spaces
- Control of partial differential equations via physics-informed neural networks
- Turnpike in optimal control of PDEs, ResNets, and beyond
- Universal regular conditional distributions via probabilistic transformers
- Stochastic integrals and Gelfand integration in Fréchet spaces
- Control on the manifolds of mappings with a view to the deep learning
- An optimal control framework for adaptive neural ODEs
- Large Sample Mean-Field Stochastic Optimization
This page was built for publication: Deep neural networks, generic universal interpolation, and controlled ODEs
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5037577)
