Distributed sequence memory of multidimensional inputs in recurrent networks

MaRDI QIDQ5361270zbMATH OpenFDO

Authors Adam S. Charles, Dong Yin, Christopher J. Rozell

Publication date 27 September 2017

Full work available at URL https://arxiv.org/abs/1605.08346, http://jmlr.csail.mit.edu/papers/v18/16-270.html

zbMATH Keywords

low-rank recoveryrecurrent neural networksrestricted isometry propertyshort-term memorysparse signal recovery

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05) Artificial neural networks and deep learning (68T07)

Abstract: Recurrent neural networks (RNNs) have drawn interest from machine learning researchers because of their effectiveness at preserving past inputs for time-varying data processing tasks. To understand the success and limitations of RNNs, it is critical that we advance our analysis of their fundamental memory properties. We focus on echo state networks (ESNs), which are RNNs with simple memoryless nodes and random connectivity. In most existing analyses, the short-term memory (STM) capacity results conclude that the ESN network size must scale linearly with the input size for unstructured inputs. The main contribution of this paper is to provide general results characterizing the STM capacity for linear ESNs with multidimensional input streams when the inputs have common low-dimensional structure: sparsity in a basis or significant statistical dependence between inputs. In both cases, we show that the number of nodes in the network must scale linearly with the information rate and poly-logarithmically with the ambient input dimension. The analysis relies on advanced applications of random matrix theory and results in explicit non-asymptotic bounds on the recovery error. Taken together, this analysis provides a significant step forward in our understanding of the STM properties in RNNs.

Recommendations

• Memory and forecasting capacities of nonlinear recurrent networks
• Short-term memory capacity in networks via the restricted isometry property
• Echo state property linked to an input: exploring a fundamental characteristic of recurrent neural networks
• Recurrent kernel machines: computing with infinite echo state networks
• scientific article; zbMATH DE number 2033224

Cited in

(10)

• Dimension reduction in recurrent networks by canonicalization
• Memory and forecasting capacities of nonlinear recurrent networks
• scientific article; zbMATH DE number 2033224 (Why is no real title available?)
• Understanding recurrent neural networks using nonequilibrium response theory
• Low-dimensional manifolds support multiplexed integrations in recurrent neural networks
• A theory of sequence indexing and working memory in recurrent neural networks
• Opening the black box: low-dimensional dynamics in high-dimensional recurrent neural networks
• Short-term memory capacity in networks via the restricted isometry property
• Linear RNNs provably learn linear dynamical systems
• A constrained regularization approach for input-driven recurrent neural networks

This page was built for publication: Distributed sequence memory of multidimensional inputs in recurrent networks

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5361270)