Deep relaxation: partial differential equations for optimizing deep neural networks
deep learningneural networksoptimal controlpartial differential equationsstochastic gradient descent
Viscosity solutions to PDEs (35D40) KdV equations (Korteweg-de Vries equations) (35Q53) Numerical optimization and variational techniques (65K10) Neural nets applied to problems in time-dependent statistical mechanics (82C32) Neural networks for/in biological studies, artificial life and related topics (92B20) Optimal stochastic control (93E20)
The output of a deep neural network is defined as \(y(x;\xi)=\sigma(x^p\sigma(x^{p-1}\dots\sigma(x^1\xi)\dots)\) which is a nested composition of linear functions depending on inputs \(\xi\in \mathbb{R}^d\) and weights \(x\in \mathbb{R}^n\), which are the parameters of the network. Performing a supervised training the goal is to minimize a certain loss function \(f(x)\). In the background part, there is a short discussion on stochastic gradient descent (SGD) and stochastic gradient descent in continuous time methods, as well as a presentation of some references. The third section concentrates on few results on the PDE-interpretation of local entropy, as a derivation of the viscous Hamilton-Jacobi PDE or the Hopf-Lax formula for the Hamilton-Jacobi equation. The fourth section deals with the derivation of local entropy via homogenization of SDEs. Further, one shows results on stochastic control for a variant of local entropy. Since it has been proved, in a previous section, that the regularized loss function is the solution of a viscous Hamilton-Jacobi equation, one can apply semi-concavity estimates from PDE theory and quantify the amount of smoothing. Some examples to illustrate the widening of local minima are presented. The last two sections are devoted to a comparison of various algorithms presented in this paper. The aim is to show that the considered collection of PDE methods improve results on modern datasets as MNIST or CIFAR datasets. All along the article, one uses intensively the published results of the authors, compares results, investigates and improves some of the details.
- Selection dynamics for deep neural networks
- Deep neural networks motivated by partial differential equations
- Optimally weighted loss functions for solving PDEs with neural networks
- Deep learning as optimal control problems: models and numerical methods
- A mean-field optimal control formulation of deep learning
- A Stochastic Approximation Method
- Adaptive subgradient methods for online learning and stochastic optimization
- Contractions in the 2-Wasserstein length space and thermalization of granular media
- Controlled Markov processes and viscosity solutions
- Convergent Difference Schemes for Degenerate Elliptic and Parabolic Equations: Hamilton--Jacobi Equations and Free Boundary Problems
- Entropy-SGD: biasing gradient descent into wide valleys
- Free energy computations. A mathematical perspective
- scientific article; zbMATH DE number 6378127 (Why is no real title available?)
- scientific article; zbMATH DE number 3850830 (Why is no real title available?)
- scientific article; zbMATH DE number 3505708 (Why is no real title available?)
- scientific article; zbMATH DE number 1273988 (Why is no real title available?)
- scientific article; zbMATH DE number 1181255 (Why is no real title available?)
- scientific article; zbMATH DE number 2121575 (Why is no real title available?)
- scientific article; zbMATH DE number 3894218 (Why is no real title available?)
- Inequalities: theory of majorization and its applications
- Large population stochastic dynamic games: closed-loop McKean-Vlasov systems and the Nash certainty equivalence principle
- Learning representations by back-propagating errors
- Local entropy as a measure for sampling solutions in constraint satisfaction problems
- Mean field games
- Minimizing finite sums with the stochastic average gradient
- Monotone Operators and the Proximal Point Algorithm
- Multiscale Methods
- Optimal transport for applied mathematicians. Calculus of variations, PDEs, and modeling
- Optimization methods for large-scale machine learning
- Principles of multiscale modeling
- Proximité et dualité dans un espace hilbertien
- Replica symmetry breaking condition exposed by random matrix calculation of landscape complexity
- Semiconcave functions, Hamilton-Jacobi equations, and optimal control
- Smoothing methods for nonsmooth, nonconvex minimization
- Stochastic First- and Zeroth-Order Methods for Nonconvex Stochastic Programming
- Stochastic processes and applications. Diffusion processes, the Fokker-Planck and Langevin equations
- The Fokker-Planck equation. Methods of solution and applications
- The Variational Formulation of the Fokker--Planck Equation
- Forward stability of ResNet and its variants
- Deep neural networks motivated by partial differential equations
- Selection dynamics for deep neural networks
- A mean field games approach to cluster analysis
- On Bayesian posterior mean estimators in imaging sciences and Hamilton-Jacobi partial differential equations
- Quantized convolutional neural networks through the lens of partial differential equations
- Laplacian smoothing gradient descent
- Distributed kernel gradient descent algorithm for minimum error entropy principle
- A regularization interpretation of the proximal point method for weakly convex functions
- Jarzynski's equality, fluctuation theorems, and variance reduction: mathematical analysis and numerical algorithms
- A multiscale neural network based on hierarchical nested bases
- Run-and-inspect method for nonconvex optimization and global optimality bounds for R-local minimizers
- Deep limits of residual neural networks
- CPINNs: a coupled physics-informed neural networks for the closed-loop geothermal system
- Stochastic control liaisons. Richard Sinkhorn meets Gaspard Monge on a Schrödinger bridge
- Neural ODEs as the deep limit of ResNets with constant weights
- Learning and meta-learning of stochastic advection–diffusion–reaction systems from sparse measurements
- A deep learning Galerkin method for the second-order linear elliptic equations
- Entropic gradient descent algorithms and wide flat minima*
- Biological population management based on a Hamilton–Jacobi–Bellman equation with boundary blow up
- Derivative-Free Bayesian Inversion Using Multiscale Dynamics
- Particle dual averaging: optimization of mean field neural network with global convergence rate analysis*
- A Continuous-Time Analysis of Distributed Stochastic Gradient
- Accelerated Optimization in the PDE Framework Formulations for the Active Contour Case
- Adaptive smoothing for path integral control
- A Multiscale Neural Network Based on Hierarchical Matrices
- BinaryRelax: a relaxation approach for training deep neural networks with quantized weights
- Entropy-SGD: biasing gradient descent into wide valleys
- The Continuous Formulation of Shallow Neural Networks as Wasserstein-Type Gradient Flows
- Deterministic neural networks optimization from a continuous and energy point of view
- Singular perturbations in stochastic optimal control with unbounded data
- Markov chain stochastic DCA and applications in deep learning with PDEs regularization
- Efficient and stable SAV-based methods for gradient flows arising from deep learning
- Global solutions to nonconvex problems by evolution of Hamilton-Jacobi PDEs
- Cell-average based neural network method for Hunter-Saxton equations
- Deep relaxation of controlled stochastic gradient descent via singular perturbations
- Entropy-based guidance of deep neural networks for accelerated convergence and improved performance
- Fast and robust consensus-based optimization via optimal feedback control
- A multiscale consensus-based algorithm for multilevel optimization
- Convergence of Markov chains for constant step-size stochastic gradient descent with separable functions
- Laplace meets Moreau: smooth approximation to infimal convolutions using Laplace's method
- Primal-dual hybrid gradient algorithms for computing time-implicit Hamilton-Jacobi equations
- Time-frequency analysis for neural networks
- Solving high-dimensional Hamilton-Jacobi-Bellman PDEs using neural networks: perspectives from the theory of controlled diffusions and measures on path space
- On large batch training and sharp minima: a Fokker-Planck perspective
Uses Software
This page was built for publication: Deep relaxation: partial differential equations for optimizing deep neural networks
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2319762)
