How does momentum benefit deep neural networks architecture design? A few case studies

DOI10.1007/s40687-022-00352-0OpenAlexW3205535720MaRDI QIDQ2079522

Yanyan Li

Publication date: 30 September 2022

Published in: Research in the Mathematical Sciences (Search for Journal in Brave)

Full work available at URL: https://arxiv.org/abs/2110.07034

Mathematics Subject Classification ID

Biology and other natural sciences (92-XX) Computer science (68-XX)

Related Items (1)

Learning proper orthogonal decomposition of complex dynamics using heavy-ball neural ODEs

Uses Software

• MNIST
• OpenAI Gym
• PyTorch
• RMSprop
• Adam
• GitHub
• Tensor2Tensor
• MuJoCo
• BERT
• PixelCNN++
• BLEU
• NeuralCDE
• torchcde
• ViT
• Music Transformer
• Synthesizer
• FMMformer
• Longformer
• Nyströmformer
• Reformer

Cites Work

• Unnamed Item
• Adversarial defense via the data-dependent activation, total variation minimization, and adversarial training
• THE HEAVY BALL WITH FRICTION METHOD, I. THE CONTINUOUS DYNAMICAL SYSTEM: GLOBAL EXPLORATION OF THE LOCAL MINIMA OF A REAL-VALUED FUNCTION BY ASYMPTOTIC ANALYSIS OF A DISSIPATIVE DYNAMICAL SYSTEM
• Stable architectures for deep neural networks
• A variational perspective on accelerated methods in optimization
• Graph interpolating activation improves both natural and robust accuracies in data-efficient deep learning
• Scheduled Restart Momentum for Accelerated Stochastic Gradient Descent
• Some methods of speeding up the convergence of iteration methods

This page was built for publication: How does momentum benefit deep neural networks architecture design? A few case studies