Dynamic neural Turing machine with continuous and discrete addressing schemes

DOI10.1162/NECO_A_01060zbMATH Open1472.68142DBLPjournals/neco/GulcehreCCB18OpenAlexW2751304263WikidataQ49579907 ScholiaQ49579907MaRDI QIDQ5157151FDOQ5157151

Authors Çağlar Gülçehre, Sarath Chandar, Yoshua Bengio, Kyung Hyun Cho

Publication date 12 October 2021

Published in Neural Computation (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1607.00036

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05) Networks and circuits as models of computation; circuit complexity (68Q06)

Abstract: We extend neural Turing machine (NTM) model into a dynamic neural Turing machine (D-NTM) by introducing a trainable memory addressing scheme. This addressing scheme maintains for each memory cell two separate vectors, content and address vectors. This allows the D-NTM to learn a wide variety of location-based addressing strategies including both linear and nonlinear ones. We implement the D-NTM with both continuous, differentiable and discrete, non-differentiable read/write mechanisms. We investigate the mechanisms and effects of learning to read and write into a memory through experiments on Facebook bAbI tasks using both a feedforward and GRUcontroller. The D-NTM is evaluated on a set of Facebook bAbI tasks and shown to outperform NTM and LSTM baselines. We have done extensive analysis of our model and different variations of NTM on bAbI task. We also provide further experimental results on sequential pMNIST, Stanford Natural Language Inference, associative recall and copy tasks.

Recommendations

• Synthesis and analysis of elementary algorithms for a differential neural computer
• Overcoming catastrophic forgetting in neural networks
• Elements for a general memory structure: properties of recurrent neural networks used to form situation models
• Improving Generalization Capabilities of Dynamic Neural Networks
• scientific article; zbMATH DE number 1928643

Cites work

• Deep learning
• Robust Estimation of a Location Parameter
• Simple statistical gradient-following algorithms for connectionist reinforcement learning

Cited in

(1)

• Synthesis and analysis of elementary algorithms for a differential neural computer

Describes a project that uses

Uses Software

• Adam
• GloVe
• Zoneout
• VQA

This page was built for publication: Dynamic neural Turing machine with continuous and discrete addressing schemes

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5157151)