Deep neural networks, generic universal interpolation, and controlled ODEs

DOI10.1137/19M1284117zbMATH Open1485.93062arXiv1908.07838OpenAlexW3089885363MaRDI QIDQ5037577FDOQ5037577

Authors: Christa Cuchiero, Martin Larsson, Josef Teichmann 

Publication date: 1 March 2022

Published in: SIAM Journal on Mathematics of Data Science (Search for Journal in Brave)

Abstract: A recent paradigm views deep neural networks as discretizations of certain controlled ordinary differential equations, sometimes called neural ordinary differential equations. We make use of this perspective to link expressiveness of deep networks to the notion of controllability of dynamical systems. Using this connection, we study an expressiveness property that we call universal interpolation, and show that it is generic in a certain sense. The universal interpolation property is slightly weaker than universal approximation, and disentangles supervised learning on finite training sets from generalization properties. We also show that universal interpolation holds for certain deep neural networks even if large numbers of parameters are left untrained, and are instead chosen randomly. This lends theoretical support to the observation that training with random initialization can be successful even when most parameters are largely unchanged through the training. Our results also explore what a minimal amount of trainable parameters in neural ordinary differential equations could be without giving up on expressiveness.

Full work available at URL: https://arxiv.org/abs/1908.07838

Recommendations

• Neural ODE Control for Classification, Approximation, and Transport
• Interpolation and approximation via momentum ResNets and neural ODEs
• Control on the manifolds of mappings with a view to the deep learning
• Deep learning as optimal control problems: models and numerical methods
• Expressivity of Deep Neural Networks

zbMATH Keywords

Lie bracketsuniversal interpolationcontrolled dynamical systemsdeep neural networksHörmander condition

Mathematics Subject Classification ID

Artificial neural networks and deep learning (68T07) Controllability (93B05) Control/observation systems governed by ordinary differential equations (93C15)

Cites Work

• A tour of subriemannian geometries, their geodesics and applications
• Approximation by superpositions of a sigmoidal function
• Title not available (Why is that?)
• Title not available (Why is that?)
• Defining relations for classical Lie algebras of polynomial vector fields.
• On Everywhere Dense Imbedding of Free Groups in Lie Groups
• Just interpolate: kernel ``ridgeless regression can generalize
• A proposal on machine learning via dynamical systems

Cited In (13)

• An optimal control framework for adaptive neural ODEs
• Large Sample Mean-Field Stochastic Optimization
• Control on the manifolds of mappings with a view to the deep learning
• Interpolation and approximation via momentum ResNets and neural ODEs
• Reinforcement learning and stochastic optimisation
• Turnpike in optimal control of PDEs, ResNets, and beyond
• Neural networks in Fréchet spaces
• Pricing options on flow forwards by neural networks in a Hilbert space
• Universal regular conditional distributions via probabilistic transformers
• Stochastic integrals and Gelfand integration in Fréchet spaces
• Statistical inference and neural network training based on stochastic difference model for air pollution and associated disease transmission
• Neural ODE Control for Classification, Approximation, and Transport
• Control of partial differential equations via physics-informed neural networks

Uses Software

• FFJORD
• torchdiffeq
• ANODEs

This page was built for publication: Deep neural networks, generic universal interpolation, and controlled ODEs

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5037577)