slimTrain---A Stochastic Approximation Method for Training Separable Deep Neural Networks

DOI10.1137/21M1452512WikidataQ114073961 ScholiaQ114073961MaRDI QIDQ5095499

Matthias Chung, Lars Ruthotto, Elizabeth Newman, Julianne Chung

Publication date: 9 August 2022

Published in: SIAM Journal on Scientific Computing (Search for Journal in Brave)

Full work available at URL: https://arxiv.org/abs/2109.14002

zbMATH Keywords

stochastic approximationinverse problemsiterative methodsdeep learningvariable projectionlearning rates

Mathematics Subject Classification ID

Artificial neural networks and deep learning (68T07) Probabilistic models, generic numerical methods in probability and statistics (65C20) Numerical methods for mathematical programming, optimization and variational techniques (65K99)

Related Items (2)

A variable projection method for large-scale inverse problems with \(\ell^1\) regularization ⋮ A method for computing inverse parametric PDE problems with random-weight neural networks

Uses Software

• LSQR
• TensorFlow
• PyTorch
• ImageNet
• AdaGrad
• Adam
• U-Net
• torchdiffeq
• AlexNet
• TensorFlow.jl
• CRAIG

Cites Work

• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• A Stochastic Quasi-Newton Method for Large-Scale Optimization
• Bayesian deep convolutional encoder-decoder networks for surrogate modeling and uncertainty quantification
• Deep neural networks motivated by partial differential equations
• Deep UQ: learning deep neural network surrogate models for high dimensional uncertainty quantification
• Model reduction and neural networks for parametric PDEs
• Derivative-informed projected neural networks for high-dimensional parametric maps governed by PDEs
• Physics-constrained deep learning for high-dimensional surrogate modeling and uncertainty quantification without labeled data
• Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations
• Coordinate descent algorithms
• Variable projection for nonlinear least squares problems
• A proposal on machine learning via dynamical systems
• Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function
• Efficiency of Coordinate Descent Methods on Huge-Scale Optimization Problems
• An Efficient Iterative Approach for Large-Scale Separable Nonlinear Inverse Problems
• Generalized Cross-Validation as a Method for Choosing a Good Ridge Parameter
• LSQR: An Algorithm for Sparse Linear Equations and Sparse Least Squares
• Practical Approximate Solutions to Linear Operator Equations When the Data are Noisy
• Separable nonlinear least squares: the variable projection method and its applications
• Randomized Quasi-Newton Updates Are Linearly Convergent Matrix Inversion Algorithms
• Stable architectures for deep neural networks
• Optimization Methods for Large-Scale Machine Learning
• Computational Methods for Inverse Problems
• Sampled Tikhonov regularization for large linear inverse problems
• Sampled limited memory methods for massive linear inverse problems
• Train Like a (Var)Pro: Efficient Training of Neural Networks with Variable Projection
• Information-Theoretic Lower Bounds on the Oracle Complexity of Stochastic Convex Optimization
• On the Convergence of Block Coordinate Descent Type Methods
• The Differentiation of Pseudo-Inverses and Nonlinear Least Squares Problems Whose Variables Separate
• Stochastic Quasi-Newton Methods for Nonconvex Stochastic Optimization
• A Stochastic Approximation Method
• Exact and inexact subsampled Newton methods for optimization
• Approximation by superpositions of a sigmoidal function

This page was built for publication: slimTrain---A Stochastic Approximation Method for Training Separable Deep Neural Networks