The echo index and multistability in input-driven recurrent neural networks

DOI10.1016/J.PHYSD.2020.132609MaRDI QIDQ2127403zbMATH OpenOpenAlexFDO

Authors Andrea Ceni, Peter Ashwin, Lorenzo Livi, Claire Postlethwaite

Publication date 20 April 2022

Published in Physica D (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2001.07694, https://hdl.handle.net/10871/122103

zbMATH Keywords

machine learningnonautonomous dynamical systemsmultistabilityinput-driven systemsrecurrent neural networksecho state property

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05) Topological dynamics of nonautonomous systems (37B55) Networks and circuits as models of computation; circuit complexity (68Q06)

Abstract: A recurrent neural network (RNN) possesses the echo state property (ESP) if, for a given input sequence, it ``forgets any internal states of the driven (nonautonomous) system and asymptotically follows a unique, possibly complex trajectory. The lack of ESP is conventionally understood as a lack of reliable behaviour in RNNs. Here, we show that RNNs can reliably perform computations under a more general principle that accounts only for their local behaviour in phase space. To this end, we formulate a generalisation of the ESP and introduce an echo index to characterise the number of simultaneously stable responses of a driven RNN. We show that it is possible for the echo index to change with inputs, highlighting a potential source of computational errors in RNNs due to characteristics of the inputs driving the dynamics.

Recommendations

• Multistability in Recurrent Neural Networks
• Delay-dependent multistability in recurrent neural networks
• Echo state property linked to an input: exploring a fundamental characteristic of recurrent neural networks
• Multistability and instability analysis of recurrent neural networks with time-varying delays
• Delay-independent exponential stability of recurrent neural networks
• Multistability of delayed recurrent neural networks with Mexican hat activation functions
• Dynamical behavior of recurrent neural networks with different external inputs

Cites work

• scientific article; zbMATH DE number 3665682 (Why is no real title available?)
• scientific article; zbMATH DE number 7164779 (Why is no real title available?)
• A Geometrical Analysis of Global Stability in Trained Feedback Networks
• Detection of generalized synchronization using echo state networks
• Echo state networks are universal
• Echo state property linked to an input: exploring a fundamental characteristic of recurrent neural networks
• Forward attraction in nonautonomous difference equations
• Limitations of pullback attractors for processes
• Morse Decomposition of Attractors for Non-autonomous Dynamical Systems
• Nonautonomous continuation of bounded solutions
• Nonautonomous dynamical systems
• Optimization and applications of echo state networks with leaky- integrator neurons
• Random fixed point theorems with an application to random differential equations in Banach spaces
• Re-visiting the echo state property
• Reservoir computing approaches to recurrent neural network training
• TOWARDS A MORSE THEORY FOR RANDOM DYNAMICAL SYSTEMS

Cited in

(8)

• Learn to synchronize, synchronize to learn
• Arbitrary sensitive transitions in recurrent neural networks
• Minimal model for reservoir computing
• Transitions in echo index and dependence on input repetitions
• Embedding information onto a dynamical system
• Echo state property linked to an input: exploring a fundamental characteristic of recurrent neural networks
• Stimulus-Driven and Spontaneous Dynamics in Excitatory-Inhibitory Recurrent Neural Networks for Sequence Representation
• Generalised synchronisations, embeddings, and approximations for continuous time reservoir computers

This page was built for publication: The echo index and multistability in input-driven recurrent neural networks

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2127403)