Gated Orthogonal Recurrent Units: On Learning to Forget

DOI10.1162/NECO_A_01174MaRDI QIDQ5154142zbMATHOpenAlexWikidataFDO

Authors Li Jing, Çağlar Gülçehre, John Peurifoy, Yi-Chen Shen, Max Tegmark, Marin Soljačić, Yoshua Bengio

Publication date 1 October 2021

Published in Neural Computation (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1706.02761

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05) Neural networks for/in biological studies, artificial life and related topics (92B20)

Abstract: We present a novel recurrent neural network (RNN) based model that combines the remembering ability of unitary RNNs with the ability of gated RNNs to effectively forget redundant/irrelevant information in its memory. We achieve this by extending unitary RNNs with a gating mechanism. Our model is able to outperform LSTMs, GRUs and Unitary RNNs on several long-term dependency benchmark tasks. We empirically both show the orthogonal/unitary RNNs lack the ability to forget and also the ability of GORU to simultaneously remember long term dependencies while forgetting irrelevant information. This plays an important role in recurrent neural networks. We provide competitive results along with an analysis of our model on many natural sequential tasks including the bAbI Question Answering, TIMIT speech spectrum prediction, Penn TreeBank, and synthetic tasks that involve long-term dependencies such as algorithmic, parenthesis, denoising and copying tasks.

Recommendations

• Sgornn: Combining scalar gates and orthogonal constraints in recurrent networks
• Gated Graph Recurrent Neural Networks
• Overcoming catastrophic forgetting in neural networks
• On duality of regularized exponential and linear forgetting
• A family of universal recurrent networks
• scientific article; zbMATH DE number 2042276
• A recalling-enhanced recurrent neural network: conjugate gradient learning algorithm and its convergence analysis
• Toward training recurrent neural networks for lifelong learning

Cited in

(6)

• A review of recurrent neural networks: LSTM cells and network architectures
• Stock market predictions using FastRNN-based model
• A homotopy gated recurrent unit for predicting high dimensional hyperchaos
• Dynamical systems as temporal feature spaces
• Divide and conquer: learning chaotic dynamical systems with multistep penalty neural ordinary differential equations
• Sgornn: Combining scalar gates and orthogonal constraints in recurrent networks

Describes a project that uses

Uses Software

• Penn Treebank
• RMSprop
• Adam
• DARPA TIMIT

This page was built for publication: Gated Orthogonal Recurrent Units: On Learning to Forget

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5154142)