Stable architectures for deep neural networks

DOI10.1088/1361-6420/AA9A90zbMath1426.68236arXiv1705.03341OpenAlexW3098011980MaRDI QIDQ4607800

Lars Ruthotto, Eldad Haber

Publication date: 14 March 2018

Published in: Inverse Problems (Search for Journal in Brave)

Full work available at URL: https://arxiv.org/abs/1705.03341

zbMATH Keywords

parameter estimationmachine learningdynamic inverse problemsimage classificationdeep neural networks

Mathematics Subject Classification ID

Artificial neural networks and deep learning (68T07) Learning and adaptive systems in artificial intelligence (68T05) Inverse problems involving ordinary differential equations (34A55)

Related Items (only showing first 100 items - show all)

On the geometric mechanics of assignment flows for metric data labeling ⋮ Odd and even functions in the design problem of new chaotic attractors ⋮ On dynamical system modeling of learned primal-dual with a linear operator \(\mathcal{K}\): stability and convergence properties ⋮ An optimal control framework for adaptive neural ODEs ⋮ A dynamical system-based framework for dimension reduction ⋮ Energy-informed graph transformer model for solid mechanical analyses ⋮ A novel recurrent neural network of gated unit based on Euler's method and application ⋮ Mini-workshop: Nonlinear approximation of high-dimensional functions in scientific computing. Abstracts from the mini-workshop held October 15--20, 2023 ⋮ A block-coordinate approach of multi-level optimization with an application to physics-informed neural networks ⋮ KAN-ODEs: Kolmogorov-Arnold network ordinary differential equations for learning dynamical systems and hidden physics ⋮ Vanilla feedforward neural networks as a discretization of dynamical systems ⋮ NINNs: Nudging induced neural networks ⋮ Predict globally, correct locally: parallel-in-time optimization of neural networks ⋮ Surrogate construction via weight parameterization of residual neural networks ⋮ A generalized framework of neural networks for Hamiltonian systems ⋮ Parameter calibration with stochastic gradient descent for interacting particle systems driven by neural networks ⋮ Interpolation and approximation via momentum ResNets and neural ODEs ⋮ EnResNet: ResNets Ensemble via the Feynman--Kac Formalism for Adversarial Defense and Beyond ⋮ Stability of Deep Neural Networks via Discrete Rough Paths ⋮ A survey on deep learning and its applications ⋮ Turnpike in optimal control of PDEs, ResNets, and beyond ⋮ ODE-RU: a dynamical system view on recurrent neural networks ⋮ Machine learning from a continuous viewpoint. I ⋮ Analytic continuation of noisy data using Adams Bashforth residual neural network ⋮ Meta-mgnet: meta multigrid networks for solving parameterized partial differential equations ⋮ Activation function design for deep networks: linearity and effective initialisation ⋮ Feasibility-based fixed point networks ⋮ Wasserstein-Based Projections with Applications to Inverse Problems ⋮ Designing rotationally invariant neural networks from PDEs and variational methods ⋮ On the regularized risk of distributionally robust learning over deep neural networks ⋮ Personalized Algorithm Generation: A Case Study in Learning ODE Integrators ⋮ A non-intrusive correction algorithm for classification problems with corrupted data ⋮ slimTrain---A Stochastic Approximation Method for Training Separable Deep Neural Networks ⋮ Large Sample Mean-Field Stochastic Optimization ⋮ Deep Neural Networks and PIDE Discretizations ⋮ A backward SDE method for uncertainty quantification in deep learning ⋮ Generative modeling via tensor train sketching ⋮ A literature survey of matrix methods for data science ⋮ Deep learning methods for partial differential equations and related parameter identification problems ⋮ Deep limits of residual neural networks ⋮ A framework for machine learning of model error in dynamical systems ⋮ Multilevel Objective-Function-Free Optimization with an Application to Neural Networks Training ⋮ Neural network representation of time integrators ⋮ Dual non-autonomous deep convolutional neural network for image denoising ⋮ Deep neural networks on diffeomorphism groups for optimal shape reparametrization ⋮ CD-ROM: complemented deep -- reduced order model ⋮ Locally-symplectic neural networks for learning volume-preserving dynamics ⋮ Deep learning approximation of diffeomorphisms via linear-control systems ⋮ Sparsity in long-time control of neural ODEs ⋮ Accuracy and architecture studies of residual neural network method for ordinary differential equations ⋮ Neural ODE Control for Classification, Approximation, and Transport ⋮ A mathematical perspective of machine learning ⋮ Unnamed Item ⋮ Physics-Informed Probabilistic Learning of Linear Embeddings of Nonlinear Dynamics with Guaranteed Stability ⋮ A functional approach to interpreting the role of the adjoint equation in machine learning ⋮ Estimating a Potential Without the Agony of the Partition Function ⋮ Multi‐fidelity data fusion through parameter space reduction with applications to automotive engineering ⋮ Connections between numerical algorithms for PDEs and neural networks ⋮ Applications of time parallelization ⋮ Relational intelligence recognition in online social networks -- a survey ⋮ Deep learning via dynamical systems: an approximation perspective ⋮ A Multilevel Method for Many-Electron Schrödinger Equations Based on the Atomic Cluster Expansion ⋮ Learning Hamiltonian systems with mono-implicit Runge-Kutta methods ⋮ Optimal control using to approximate probability distribution of observation set ⋮ Numerical Analysis for Convergence of a Sample-Wise Backpropagation Method for Training Stochastic Neural Networks ⋮ Convex and concave envelopes of artificial neural network activation functions for deterministic global optimization ⋮ On mathematical modeling in image reconstruction and beyond ⋮ Unnamed Item ⋮ An artificial neural network framework for reduced order modeling of transient flows ⋮ PDE-Net 2.0: learning PDEs from data with a numeric-symbolic hybrid deep network ⋮ Generalization of partitioned Runge-Kutta methods for adjoint systems ⋮ Mini-workshop: Deep learning and inverse problems. Abstracts from the mini-workshop held March 4--10, 2018 ⋮ Dynamic inverse problems: modelling—regularization—numerics ⋮ State-Space Representations of Deep Neural Networks ⋮ Forward stability of ResNet and its variants ⋮ Deep neural networks motivated by partial differential equations ⋮ Residual networks as flows of diffeomorphisms ⋮ Variational networks: an optimal control approach to early stopping variational methods for image restoration ⋮ Networks for nonlinear diffusion problems in imaging ⋮ ADMM-softmax: an ADMM approach for multinomial logistic regression ⋮ Unnamed Item ⋮ PNKH-B: A Projected Newton--Krylov Method for Large-Scale Bound-Constrained Optimization ⋮ Train Like a (Var)Pro: Efficient Training of Neural Networks with Variable Projection ⋮ How Deep Are Deep Gaussian Processes? ⋮ Selection dynamics for deep neural networks ⋮ Unnamed Item ⋮ Efficient and stable SAV-based methods for gradient flows arising from deep learning ⋮ Learning adaptive regularization for image labeling using geometric assignment ⋮ Continuous-time system identification with neural networks: model structures and fitting criteria ⋮ An Efficient Parallel-in-Time Method for Optimization with Parabolic PDEs ⋮ On Functions Computed on Trees ⋮ Deep learning as optimal control problems: models and numerical methods ⋮ Model reduction and neural networks for parametric PDEs ⋮ The Random Feature Model for Input-Output Maps between Banach Spaces ⋮ Solving inverse problems using data-driven models ⋮ A sequential quadratic Hamiltonian algorithm for training explicit RK neural networks ⋮ A mean-field optimal control formulation of deep learning ⋮ Time-series learning of latent-space dynamics for reduced-order model closure ⋮ How does momentum benefit deep neural networks architecture design? A few case studies ⋮ Quantized convolutional neural networks through the lens of partial differential equations

Uses Software

• MNIST
• ImageNet
• FAIR.m
• AlexNet

Cites Work

• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Sample size selection in optimization methods for machine learning
• The cascadic multigrid method for elliptic problems
• Sub-sampled Newton methods
• A GCV based method for nonlinear ill-posed problems
• Nonlinear inverse scale space methods
• On the Use of Stochastic Hessian Information in Optimization Methods for Machine Learning
• Computational Optimization of Systems Governed by Partial Differential Equations
• A Regularized Gauss–Newton Trust Region Approach to Imaging in Diffuse Optical Tomography
• Learning Deep Architectures for AI
• FAIR
• Numerical Optimization
• Rank-Deficient and Discrete Ill-Posed Problems
• Adjoint Sensitivity Analysis for Differential-Algebraic Equations: The Adjoint DAE System and Its Numerical Solution
• Optimization Methods for Large-Scale Machine Learning
• Learning representations by back-propagating errors
• Priorconditioners for linear systems
• Methods of conjugate gradients for solving linear systems
• A Stochastic Approximation Method
• The elements of statistical learning. Data mining, inference, and prediction

This page was built for publication: Stable architectures for deep neural networks