A Simple and Efficient Tensor Calculus for Machine Learning
From MaRDI portal
Abstract: Computing derivatives of tensor expressions, also known as tensor calculus, is a fundamental task in machine learning. A key concern is the efficiency of evaluating the expressions and their derivatives that hinges on the representation of these expressions. Recently, an algorithm for computing higher order derivatives of tensor expressions like Jacobians or Hessians has been introduced that is a few orders of magnitude faster than previous state-of-the-art approaches. Unfortunately, the approach is based on Ricci notation and hence cannot be incorporated into automatic differentiation frameworks from deep learning like TensorFlow, PyTorch, autograd, or JAX that use the simpler Einstein notation. This leaves two options, to either change the underlying tensor representation in these frameworks or to develop a new, provably correct algorithm based on Einstein notation. Obviously, the first option is impractical. Hence, we pursue the second option. Here, we show that using Ricci notation is not necessary for an efficient tensor calculus and develop an equally efficient method for the simpler Einstein notation. It turns out that turning to Einstein notation enables further improvements that lead to even better efficiency. The methods that are described in this paper have been implemented in the online tool www.MatrixCalculus.org for computing derivatives of matrix and tensor expressions. An extended abstract of this paper appeared as "A Simple and Efficient Tensor Calculus", AAAI 2020.
Recommendations
- Tensor networks in machine learning
- Tensor Learning for Regression
- Tensor Regression
- A Riemannian framework for tensor computing
- Learning with tensors: a framework based on convex optimization and spectral regularization
- Theoretical and experimental analyses of tensor-based regression and classification
- scientific article; zbMATH DE number 7255154
Cites work
- scientific article; zbMATH DE number 3144846 (Why is no real title available?)
- scientific article; zbMATH DE number 4074523 (Why is no real title available?)
- scientific article; zbMATH DE number 6982909 (Why is no real title available?)
- scientific article; zbMATH DE number 845714 (Why is no real title available?)
- 10.1162/jmlr.2003.3.4-5.993
- A Fast Learning Algorithm for Deep Belief Nets
- Collected matrix derivative results for forward and reverse mode algorithmic differentiation
- Efficient computation of sparse hessians using coloring and automatic differentiation
- Evaluating Derivatives
- Learning representations by back-propagating errors
- On the implementation of automatic differentiation tools
- Optimal Jacobian accumulation is NP-complete
- Optimal accumulation of Jacobian matrices by elimination methods on the dual computational graph
- Support-vector networks
- The Tapenade automatic differentiation tool, principles, model, and specification
Cited in
(3)
Describes a project that uses
Uses Software
This page was built for publication: A Simple and Efficient Tensor Calculus for Machine Learning
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q4988921)
