Semi-implicit back propagation

DOI10.23952/JNVA.7.2023.4.08MaRDI QIDQ6085636zbMATH OpenFDO

Authors Xiaoqun Zhang

Publication date 12 December 2023

Published in Journal of Nonlinear and Variational Analysis (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2002.03516

zbMATH Keywords

back propagationneural networkproximal mappingSGD

Mathematics Subject Classification ID

Artificial neural networks and deep learning (68T07)

Abstract: Neural network has attracted great attention for a long time and many researchers are devoted to improve the effectiveness of neural network training algorithms. Though stochastic gradient descent (SGD) and other explicit gradient-based methods are widely adopted, there are still many challenges such as gradient vanishing and small step sizes, which leads to slow convergence and instability of SGD algorithms. Motivated by error back propagation (BP) and proximal methods, we propose a semi-implicit back propagation method for neural network training. Similar to BP, the difference on the neurons are propagated in a backward fashion and the parameters are updated with proximal mapping. The implicit update for both hidden neurons and parameters allows to choose large step size in the training algorithm. Finally, we also show that any fixed point of convergent sequences produced by this algorithm is a stationary point of the objective loss function. The experiments on both MNIST and CIFAR-10 demonstrate that the proposed semi-implicit BP algorithm leads to better performance in terms of both loss decreasing and training/validation accuracy, compared to SGD and a similar algorithm ProxBP.

Recommendations

• Numerical Analysis for Convergence of a Sample-Wise Backpropagation Method for Training Stochastic Neural Networks
• scientific article; zbMATH DE number 5524483
• Backpropagation
• Backpropagation and stochastic gradient descent method
• On the acceleration of the backpropagation training method

Cites work

• A block coordinate descent method for regularized multiconvex optimization with applications to nonnegative tensor factorization and completion
• A Stochastic Approximation Method
• Adaptive subgradient methods for online learning and stochastic optimization
• scientific article; zbMATH DE number 7306852 (Why is no real title available?)
• Layer-Parallel Training of Deep Residual Neural Networks
• Learning representations by back-propagating errors
• Natural language processing (almost) from scratch
• On gradients of functions definable in o-minimal structures
• On semi- and subanalytic geometry
• Sketch-based empirical natural gradient methods for deep learning

Cited in

(1)

• AI*IA 2003: Advances in Artificial Intelligence

This page was built for publication: Semi-implicit back propagation

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6085636)