SignReLU neural network and its approximation ability
From MaRDI portal
Abstract: Deep neural networks (DNNs) have garnered significant attention in various fields of science and technology in recent years. Activation functions define how neurons in DNNs process incoming signals for them. They are essential for learning non-linear transformations and for performing diverse computations among successive neuron layers. In the last few years, researchers have investigated the approximation ability of DNNs to explain their power and success. In this paper, we explore the approximation ability of DNNs using a different activation function, called SignReLU. Our theoretical results demonstrate that SignReLU networks outperform rational and ReLU networks in terms of approximation performance. Numerical experiments are conducted comparing SignReLU with the existing activations such as ReLU, Leaky ReLU, and ELU, which illustrate the competitive practical performance of SignReLU.
Recommendations
- Deep Neural Networks with ReLU-Sine-Exponential Activations Break Curse of Dimensionality in Approximation on Hölder Class
- Neural networks with ReLU powers need less depth
- Nonlinear approximation and (deep) ReLU networks
- ReLU networks are universal approximators via piecewise linear or constant functions
- Error bounds for approximations with deep ReLU networks
Cites work
- L^\infty -Multivariate Approximation Theory
- Approximation by Combinations of ReLU and Squared ReLU Ridge Functions With <inline-formula> <tex-math notation="LaTeX">$\ell^1$ </tex-math> </inline-formula> and <inline-formula>
- Approximation of functions from korobov spaces by deep convolutional neural networks
- Approximation properties of a multilayered feedforward artificial neural network
- Approximations with deep neural networks in Sobolev time-space
- CMB data analysis and sparsity
- Deep distributed convolutional neural networks: universality
- Deep network approximation characterized by number of neurons
- Deep Network Approximation for Smooth Functions
- Deep neural networks for rotation-invariance approximation and learning
- Error bounds for approximations with deep ReLU networks
- Error bounds for approximations with deep ReLU neural networks in \(W^{s , p}\) norms
- scientific article; zbMATH DE number 713342 (Why is no real title available?)
- Learning theory estimates via integral operators and their approximations
- New error bounds for deep ReLU networks using sparse grids
- Optimal approximation of piecewise smooth functions using deep ReLU neural networks
- Optimal approximation with sparsely connected deep neural networks
- Optimal nonlinear approximation
- Universal approximation bounds for superpositions of a sigmoidal function
- Universality of deep convolutional neural networks
Cited in
(8)- Quaternionic convolutional neural networks with trainable Bessel activation functions
- Neural networks with ReLU powers need less depth
- Improving the expressive power of deep neural networks through integral activation transform
- Approximating functions with additive ridge features by deep \(\mathrm{ReLU}^k\) convolutional neural networks
- Sperf: a trainable smooth non-monotonic activation function with enhanced nonlinearity
- Approximation by weighted Durrmeyer-type max-product neural network operators
- ReLU-like non-monotonic smooth activation functions based on regularized Heaviside functions and extensions
- Construction and approximation properties of exact neural network interpolation operators activated by entire functions
This page was built for publication: SignReLU neural network and its approximation ability
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6126040)
