A homotopy training algorithm for fully connected neural networks
From MaRDI portal
Abstract: In this paper, we present a Homotopy Training Algorithm (HTA) to solve optimization problems arising from fully connected neural networks with complicated structures. The HTA dynamically builds the neural network starting from a simplified version to the fully connected network via adding layers and nodes adaptively. Therefore, the corresponding optimization problem is easy to solve at the beginning and connects to the original model via a continuous path guided by the HTA, which provides a high probability to get a global minimum. By gradually increasing the complexity of model along the continuous path, the HTA gets a rather good solution to the original loss function. This is confirmed by various numerical results including VGG models on CIFAR-10. For example, on the VGG13 model with batch normalization, HTA reduces the error rate by 11.86% on test dataset comparing with the traditional method. Moreover, the HTA also allows us to find the optimal structure for a fully connected neural network by building the neutral network adaptively.
Recommendations
Cites work
- A homotopy method based on WENO schemes for solving steady state problems of hyperbolic conservation laws
- A homotopy method for parameter estimation of nonlinear differential equations with multiple optima
- A three-dimensional steady-state tumor system
- Adaptive Multiprecision Path Tracking
- An equation-by-equation method for solving the multidimensional moment constrained maximum entropy problem
- Bifurcation for a free boundary problem modeling the growth of a tumor with a necrotic core
- BinaryRelax: a relaxation approach for training deep neural networks with quantized weights
- Computing all solutions to polynomial systems using homotopy continuation
- Convergence of a homotopy finite element method for computing steady states of Burgers' equation
- Deep learning
- Efficient path tracking methods
- scientific article; zbMATH DE number 3321507 (Why is no real title available?)
- Mathematical model of sarcoidosis
- Multilayer feedforward networks are universal approximators
- Optimization methods for large-scale machine learning
- Parameter estimation of social forces in pedestrian dynamics models via a probabilistic method
- Quantifying predictability through information theory: small sample estimation in a non-Gaussian framework
- Singular solutions, repeated roots and completeness for higher-spin chains
- Two-level spectral methods for nonlinear elliptic equations with multiple solutions
Cited in
(11)- A homotopy method for training neural networks
- A stochastic homotopy tracking algorithm for parametric systems of nonlinear equations
- A gradient descent method for solving a system of nonlinear equations
- A weight initialization based on the linear product structure for neural networks
- Greedy training algorithms for neural networks and applications to PDEs
- Greedy randomized sampling nonlinear Kaczmarz methods
- A residual-based weighted nonlinear Kaczmarz method for solving nonlinear systems of equations
- A class of pseudoinverse-free greedy block nonlinear Kaczmarz methods for nonlinear systems of equations
- Greedy capped nonlinear Kaczmarz methods
- Homotopy relaxation training algorithms for infinite-width two-layer ReLU neural networks
- Solving systems of nonlinear equations combining radial basis function neural network and Newton's method
This page was built for publication: A homotopy training algorithm for fully connected neural networks
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5160822)
