Noise Impact on a Recurrent Neural Network with a Linear Activation Function

DOI10.20537/ND230502MaRDI QIDQ6078992zbMATH OpenOpenAlexFDO

Authors Nadezhda I. Semenova

Publication date 28 September 2023

Published in Nelineinaya Dinamika (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2303.13262, http://mathnet.ru/eng/nd853

zbMATH Keywords

artificial neural networksdispersionecho state networknoiserecurrent neural networkstatisticwhite Gaussian noise

Mathematics Subject Classification ID

White noise theory (60H40) Neural nets and related approaches to inference from stochastic processes (62M45)

Abstract: In recent years, more and more researchers in the field of neural networks are interested in creating hardware implementations where neurons and the connection between them are realized physically. The physical implementation of ANN fundamentally changes the features of noise influence. In the case hardware ANNs, there are many internal sources of noise with different properties. The purpose of this paper is to study the peculiarities of internal noise propagation in recurrent ANN on the example of echo state network (ESN), to reveal ways to suppress such noises and to justify the stability of networks to some types of noises. In this paper we analyse ESN in presence of uncorrelated additive and multiplicative white Gaussian noise. Here we consider the case when artificial neurons have linear activation function with different slope coefficients. Starting from studying only one noisy neuron we complicate the problem by considering how the input signal and the memory property affect the accumulation of noise in ESN. In addition, we consider the influence of the main types of coupling matrices on the accumulation of noise. So, as such matrices, we take a uniform matrix and a diagonal-like matrices with different coefficients called "blurring" coefficient. We have found that the general view of variance and signal-to-noise ratio of ESN output signal is similar to only one neuron. The noise is less accumulated in ESN with diagonal reservoir connection matrix with large "blurring" coefficient. Especially it concerns uncorrelated multiplicative noise.

Recommendations

• Fundamental aspects of noise in analog-hardware neural networks
• Re-visiting the echo state property
• Influence of noise on the function of a physiological neural network
• The asymptotic performance of linear echo state neural networks
• Echo state network activation function based on bistable stochastic resonance

Cites work

• All-optical machine learning using diffractive deep neural networks
• Distributed Consensus Algorithms in Sensor Networks With Imperfect Communication: Link Failures and Channel Noise
• Fundamental aspects of noise in analog-hardware neural networks
• Reservoir computing approaches to recurrent neural network training

This page was built for publication: Noise Impact on a Recurrent Neural Network with a Linear Activation Function

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6078992)