Resonator networks. I: An efficient solution for factoring high-dimensional, distributed representations of data structures

DOI10.1162/NECO_A_01331zbMATH Open1497.68178DBLPjournals/neco/FradyKOS20arXiv2007.03748OpenAlexW3094277857WikidataQ100745207 ScholiaQ100745207MaRDI QIDQ3386435FDOQ3386435

Authors: E. Paxon Frady, Spencer J. Kent, Bruno A. Olshausen, Friedrich T. Sommer 

Publication date: 4 January 2021

Published in: Neural Computation (Search for Journal in Brave)

Abstract: The ability to encode and manipulate data structures with distributed neural representations could qualitatively enhance the capabilities of traditional neural networks by supporting rule-based symbolic reasoning, a central property of cognition. Here we show how this may be accomplished within the framework of Vector Symbolic Architectures (VSA) (Plate, 1991; Gayler, 1998; Kanerva, 1996), whereby data structures are encoded by combining high-dimensional vectors with operations that together form an algebra on the space of distributed representations. In particular, we propose an efficient solution to a hard combinatorial search problem that arises when decoding elements of a VSA data structure: the factorization of products of multiple code vectors. Our proposed algorithm, called a resonator network, is a new type of recurrent neural network that interleaves VSA multiplication operations and pattern completion. We show in two examples -- parsing of a tree-like data structure and parsing of a visual scene -- how the factorization problem arises and how the resonator network can solve it. More broadly, resonator networks open the possibility to apply VSAs to myriad artificial intelligence problems in real-world domains. A companion paper (Kent et al., 2020) presents a rigorous analysis and evaluation of the performance of resonator networks, showing it out-performs alternative approaches.

Full work available at URL: https://arxiv.org/abs/2007.03748

Recommendations

• Resonator Networks, 2: Factorization Performance and Capacity Compared to Optimization-Based Methods
• Representing objects, relations, and sequences
• Vector-derived transformation binding: an improved binding operation for deep symbol-like processing in neural networks
• scientific article; zbMATH DE number 1784866
• Linear recursive distributed representations

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05) Artificial neural networks and deep learning (68T07) Data structures (68P05) Neural networks for/in biological studies, artificial life and related topics (92B20) Machine vision and scene understanding (68T45) Networks and circuits as models of computation; circuit complexity (68Q06)

Cites Work

• The concentration of measure phenomenon
• Neural networks and physical systems with emergent collective computational abilities
• Randomly connected sigma–pi neurons can form associator networks
• Binding and normalization of binary sparse distributed representations by context-dependent thinning
• Tensor product variable binding and the representation of symbolic structures in connectionist systems
• Learning to represent spatial transformations with factored higher-order Boltzmann machines
• Title not available (Why is that?)
• Resonator Networks, 2: Factorization Performance and Capacity Compared to Optimization-Based Methods
• A theory of sequence indexing and working memory in recurrent neural networks

Cited In (3)

• Vector-derived transformation binding: an improved binding operation for deep symbol-like processing in neural networks
• Resonator Networks, 2: Factorization Performance and Capacity Compared to Optimization-Based Methods
• Representing objects, relations, and sequences

Uses Software

• MNIST
• Tensor2Tensor
• BERT

This page was built for publication: Resonator networks. I: An efficient solution for factoring high-dimensional, distributed representations of data structures

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q3386435)