A Differential Geometry Perspective on Orthogonal Recurrent Models

arXiv2102.09589MaRDI QIDQ6360992FDOQ6360992

Omri Azencot, N. Benjamin Erichson, Michael W. Mahoney, Mirela Ben-Chen

Publication date: 18 February 2021

Abstract: Recently, orthogonal recurrent neural networks (RNNs) have emerged as state-of-the-art models for learning long-term dependencies. This class of models mitigates the exploding and vanishing gradients problem by design. In this work, we employ tools and insights from differential geometry to offer a novel perspective on orthogonal RNNs. We show that orthogonal RNNs may be viewed as optimizing in the space of divergence-free vector fields. Specifically, based on a well-known result in differential geometry that relates vector fields and linear operators, we prove that every divergence-free vector field is related to a skew-symmetric matrix. Motivated by this observation, we study a new recurrent model, which spans the entire space of vector fields. Our method parameterizes vector fields via the directional derivatives of scalar functions. This requires the construction of latent inner product, gradient, and divergence operators. In comparison to state-of-the-art orthogonal RNNs, our approach achieves comparable or better results on a variety of benchmark tasks.

This page was built for publication: A Differential Geometry Perspective on Orthogonal Recurrent Models

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6360992)