A homotopy training algorithm for fully connected neural networks
From MaRDI portal
Publication:5160822
DOI10.1098/RSPA.2019.0662zbMATH Open1472.68131arXiv1903.09872OpenAlexW3103609633WikidataQ91866738 ScholiaQ91866738MaRDI QIDQ5160822FDOQ5160822
Authors: Qipin Chen, Wenrui Hao 
Publication date: 29 October 2021
Published in: Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences (Search for Journal in Brave)
Abstract: In this paper, we present a Homotopy Training Algorithm (HTA) to solve optimization problems arising from fully connected neural networks with complicated structures. The HTA dynamically builds the neural network starting from a simplified version to the fully connected network via adding layers and nodes adaptively. Therefore, the corresponding optimization problem is easy to solve at the beginning and connects to the original model via a continuous path guided by the HTA, which provides a high probability to get a global minimum. By gradually increasing the complexity of model along the continuous path, the HTA gets a rather good solution to the original loss function. This is confirmed by various numerical results including VGG models on CIFAR-10. For example, on the VGG13 model with batch normalization, HTA reduces the error rate by 11.86% on test dataset comparing with the traditional method. Moreover, the HTA also allows us to find the optimal structure for a fully connected neural network by building the neutral network adaptively.
Full work available at URL: https://arxiv.org/abs/1903.09872
Recommendations
machine learningartificial intelligenceneural networkhomotopy methodcomputational mathematicstraining algorithm
Cites Work
- Efficient path tracking methods
- Deep learning
- Multilayer feedforward networks are universal approximators
- Title not available (Why is that?)
- Computing all solutions to polynomial systems using homotopy continuation
- Mathematical model of sarcoidosis
- A homotopy method based on WENO schemes for solving steady state problems of hyperbolic conservation laws
- Adaptive Multiprecision Path Tracking
- Singular solutions, repeated roots and completeness for higher-spin chains
- Bifurcation for a free boundary problem modeling the growth of a tumor with a necrotic core
- A three-dimensional steady-state tumor system
- Quantifying predictability through information theory: small sample estimation in a non-Gaussian framework
- Parameter estimation of social forces in pedestrian dynamics models via a probabilistic method
- Optimization methods for large-scale machine learning
- An equation-by-equation method for solving the multidimensional moment constrained maximum entropy problem
- A homotopy method for parameter estimation of nonlinear differential equations with multiple optima
- Two-level spectral methods for nonlinear elliptic equations with multiple solutions
- BinaryRelax: a relaxation approach for training deep neural networks with quantized weights
- Convergence of a homotopy finite element method for computing steady states of Burgers' equation
Cited In (9)
- Greedy randomized sampling nonlinear Kaczmarz methods
- A residual-based weighted nonlinear Kaczmarz method for solving nonlinear systems of equations
- A class of pseudoinverse-free greedy block nonlinear Kaczmarz methods for nonlinear systems of equations
- Greedy capped nonlinear Kaczmarz methods
- A stochastic homotopy tracking algorithm for parametric systems of nonlinear equations
- Homotopy relaxation training algorithms for infinite-width two-layer ReLU neural networks
- A weight initialization based on the linear product structure for neural networks
- Greedy training algorithms for neural networks and applications to PDEs
- A gradient descent method for solving a system of nonlinear equations
This page was built for publication: A homotopy training algorithm for fully connected neural networks
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5160822)
