Deep learning via dynamical systems: an approximation perspective

DOI10.4171/JEMS/1221MaRDI QIDQ6172665zbMATH OpenFDO

Authors Qianxiao Li, Ting Lin, Zuowei Shen

Publication date 20 July 2023

Published in Journal of the European Mathematical Society (JEMS) (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1912.10382

zbMATH Keywords

approximation theorycontroldeep learningdeep neural networksdynamical systems

Mathematics Subject Classification ID

Dynamical systems in control (37N35) Approximations and expansions (41A99) Artificial neural networks and deep learning (68T07)

Abstract: We build on the dynamical systems approach to deep learning, where deep residual networks are idealized as continuous-time dynamical systems, from the approximation perspective. In particular, we establish general sufficient conditions for universal approximation using continuous-time deep residual networks, which can also be understood as approximation theories in $L^p$ using flow maps of dynamical systems. In specific cases, rates of approximation in terms of the time horizon are also established. Overall, these results reveal that composition function approximation through flow maps present a new paradigm in approximation theory and contributes to building a useful mathematical framework to investigate deep learning.

Review text

An important challenge in recent trends in deep learning is to develop a framework for its effectiveness by capturing the effect of sequential function composition in deep neural networks and one important tool used in this regard is a dynamical systems approach whereby one thinks of deep neural networks as a discretization of certain ordinary differential equations. In this interesting paper, the authors study the dynamical systems approach whereby deep residual networks are idealized as continuous-time dynamical systems, from an approximation perspective. The authors prove several theorems which give sufficient conditions for universal approximation using continuous time deep residual networks which can also be understood as approximation theories in \(L_p\) using certain flow maps of dynamical systems. The authors also establish rates of approximation in terms of some time horizons. This is a new and interesting idea in approximation theory, namely that composition function approximation through flow maps contributes to building useful mathematical frameworks to investigate deep learning. The paper is well written with ample references covering both classical and newer work on deep neural networks, their connections to approximation theory, control and differential equations.

Recommendations

• Selection dynamics for deep neural networks
• Dynamical Systems andOptimal Control Approach to Deep Learning
• Universal approximation properties for an ODENet and a ResNet: mathematical analysis and numerical experiments
• Mean-field limits of trained weights in deep learning: a dynamical systems perspective

Cites work

• \(L^p\) approximation of maps by diffeomorphisms
• A mean-field optimal control formulation of deep learning
• A priori estimates of the population risk for two-layer neural networks
• A proposal on machine learning via dynamical systems
• Affine systems in \(L_ 2(\mathbb{R}^d)\): The analysis of the analysis operator
• Controllability of nonlinear systems in Banach spaces: a survey
• Controllability questions for nonlinear systems in abstract spaces
• Deep distributed convolutional neural networks: universality
• Deep network approximation characterized by number of neurons
• Deep Network Approximation for Smooth Functions
• Deep neural networks motivated by partial differential equations
• Embedding diffeomorphisms in flows in Banach spaces
• EnResNet: ResNets Ensemble via the Feynman--Kac Formalism for Adversarial Defense and Beyond
• Error bounds for approximations with deep ReLU networks
• Error bounds for approximations with deep ReLU neural networks in \(W^{s , p}\) norms
• scientific article; zbMATH DE number 3772552 (Why is no real title available?)
• scientific article; zbMATH DE number 46308 (Why is no real title available?)
• scientific article; zbMATH DE number 53382 (Why is no real title available?)
• scientific article; zbMATH DE number 1215245 (Why is no real title available?)
• scientific article; zbMATH DE number 1380579 (Why is no real title available?)
• scientific article; zbMATH DE number 2103273 (Why is no real title available?)
• scientific article; zbMATH DE number 5180707 (Why is no real title available?)
• Maximum principle based algorithms for deep learning
• Nonlinear approximation and (deep) ReLU networks
• Nonlinear approximation via compositions
• Optimal approximation rate of ReLU networks in terms of width and depth
• Relaxed variational problems
• Selection dynamics for deep neural networks
• Stable architectures for deep neural networks
• Ten Lectures on Wavelets
• The Embedding of Homeomorphisms in Flows
• Transport analysis of infinitely deep neural network
• Variational networks: an optimal control approach to early stopping variational methods for image restoration
• Vector fields generate few diffeomorphisms

Cited in

(37)

• DANTE: deep alternations for training neural networks
• Learning dynamical systems from data: a simple cross-validation perspective. I: Parametric kernel flows
• On generalized residual network for deep learning of unknown dynamical systems
• Deep learning nonlinear multiscale dynamic problems using Koopman operator
• Computing the invariant distribution of randomly perturbed dynamical systems using deep learning
• A proposal on machine learning via dynamical systems
• Riddled basins of the optimal states in learning dynamical systems
• scientific article; zbMATH DE number 65825 (Why is no real title available?)
• Deep Network Approximation for Smooth Functions
• Deep neural networks can stably solve high-dimensional, noisy, non-linear inverse problems
• Universal approximation properties for an ODENet and a ResNet: mathematical analysis and numerical experiments
• Neural ODE Control for Classification, Approximation, and Transport
• Universal approximation on metric spaces
• Dynamical Systems–Based Neural Networks
• An optimal time variable learning framework for deep neural networks
• Designing universal causal deep learning models: The geometric (Hyper)transformer
• On mathematical modeling in image reconstruction and beyond
• On dynamical system modeling of learned primal-dual with a linear operator \(\mathcal{K}\): stability and convergence properties
• An optimal control framework for adaptive neural ODEs
• Differential equations for continuous-time deep learning
• A hybrid Sobolev gradient method for learning NODEs
• Vanilla feedforward neural networks as a discretization of dynamical systems
• Training neural networks from an ergodic perspective
• Deep networks for system identification: a survey
• On the universal approximation property of deep fully convolutional neural networks
• Constructive approximate transport maps with normalizing flows
• Clustering in pure-attention hardmax transformers and its role in sentiment analysis
• Diffeomorphic measure matching with kernels for generative modeling
• Local manifold approximation of dynamical system based on neural ordinary differential equation
• Cluster-based classification with neural ODEs via control
• A mathematical perspective on transformers
• Interpolation, approximation, and controllability of deep neural networks
• Auslander-Yorke dichotomy and its generalizations for non-autonomous dynamical systems
• PDE models for deep neural networks: learning theory, calculus of variations and optimal control
• Approximation of solutions of parameter-dependent problems by residual neural networks
• A deep layer limit analysis of transformer
• Some super-approximation rates of ReLU neural networks for Korobov functions

This page was built for publication: Deep learning via dynamical systems: an approximation perspective

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6172665)