A survey on modern trainable activation functions
From MaRDI portal
Publication:6078705
Abstract: In neural networks literature, there is a strong interest in identifying and defining activation functions which can improve neural network performance. In recent years there has been a renovated interest of the scientific community in investigating activation functions which can be trained during the learning process, usually referred to as "trainable", "learnable" or "adaptable" activation functions. They appear to lead to better network performance. Diverse and heterogeneous models of trainable activation function have been proposed in the literature. In this paper, we present a survey of these models. Starting from a discussion on the use of the term "activation function" in literature, we propose a taxonomy of trainable activation functions, highlight common and distinctive proprieties of recent and past models, and discuss main advantages and limitations of this type of approach. We show that many of the proposed approaches are equivalent to adding neuron layers which use fixed (non-trainable) activation functions and some simple local rule that constraints the corresponding weight layers.
Recommendations
- scientific article; zbMATH DE number 683527
- Activation function design for deep networks: linearity and effective initialisation
- scientific article; zbMATH DE number 1304395
- Deep Neural Networks With Trainable Activations and Controlled Lipschitz Constant
- A new class of activation functions. Some related problems and applications
Cites work
- A class \(+1\) sigmoidal activation functions for FFANNs
- Approximation by superpositions of a sigmoidal function
- Deep learning
- Designing Structured Sparse Dictionaries for Sparse Representation Modeling
- scientific article; zbMATH DE number 6378127 (Why is no real title available?)
- scientific article; zbMATH DE number 51537 (Why is no real title available?)
- scientific article; zbMATH DE number 1405266 (Why is no real title available?)
- Local minima in hierarchical structures of complex-valued neural networks
- Multilayer feedforward networks are universal approximators
- Neural network with unbounded activation functions is universal approximator
- Pattern recognition and machine learning.
Cited in
(25)- Deep neural networks with a set of node-wise varying activation functions
- Discontinuous neural networks and discontinuity learning
- Locally adaptive activation functions with slope recovery for deep and physics-informed neural networks
- Navigation system-based routing strategies in traffic flows on networks
- Tree-structured neural networks: spatiotemporal dynamics and optimal control
- Physics-informed deep learning for three-dimensional transient heat transfer analysis of functionally graded materials
- Quaternionic convolutional neural networks with trainable Bessel activation functions
- Modeling of the thermal softening of metals under impact loads and their temperature-time correspondence
- I-FENN with temporal convolutional networks: expediting the load-history analysis of non-local gradient damage propagation
- An efficient artificial neural network algorithm for solving boundary integral equations in elasticity
- Qualitative analysis of high-dimensional neural networks with three-layer structure and mutiple delays
- Smoothing piecewise linear activation functions based on mollified square root functions
- Dynamic analysis of a generalized attention deficit disorder model with soboleva activation functions
- Non-linearity and the distribution of market-based loss rates
- Learnable activation functions in physics-informed neural networks for solving partial differential equations
- Exploring the Bayes-oriented noise injection approach in neural networks
- Generalized pair-wise logit dynamic and its connection to a mean field game: theoretical and computational investigations focusing on resource management
- Some mathematical properties of flexible hyperbolic tangent activation function with application to deep neural networks
- Controlled learning of pointwise nonlinearities in neural-network-like architectures
- Unification of popular artificial neural network activation functions
- Inverse material design using deep reinforcement learning and homogenization
- A bio-inspired minimal model for non-stationary K-armed bandits
- Graph-instructed neural networks for sparse grid-based discontinuity detectors
- Neural solver for Wasserstein geodesics and optimal transport dynamics
- Information-driven nonlinear quantum neuron
This page was built for publication: A survey on modern trainable activation functions
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6078705)
