Approximation theory of tree tensor networks: tensorized univariate functions

DOI10.1007/S00365-023-09620-WarXiv2007.00118MaRDI QIDQ6076973FDOQ6076973

Authors: Mazen Ali, A. Nouy 

Publication date: 17 October 2023

Published in: Constructive Approximation (Search for Journal in Brave)

Abstract: We study the approximation of functions by tensor networks (TNs). We show that Lebesgue $L^p$-spaces in one dimension can be identified with tensor product spaces of arbitrary order through tensorization. We use this tensor product structure to define subsets of $L^p$ of rank-structured functions of finite representation complexity. These subsets are then used to define different approximation classes of tensor networks, associated with different measures of complexity. These approximation classes are shown to be quasi-normed linear spaces. We study some elementary properties and relationships of said spaces. In part II of this work, we will show that classical smoothness (Besov) spaces are continuously embedded into these approximation classes. We will also show that functions in these approximation classes do not possess any Besov smoothness, unless one restricts the depth of the tensor networks. The results of this work are both an analysis of the approximation spaces of TNs and a study of the expressivity of a particular type of neural networks (NN) -- namely feed-forward sum-product networks with sparse architecture. The input variables of this network result from the tensorization step, interpreted as a particular featuring step which can also be implemented with a neural network with a specific architecture. We point out interesting parallels to recent results on the expressivity of rectified linear unit (ReLU) networks -- currently one of the most popular type of NNs.

Full work available at URL: https://arxiv.org/abs/2007.00118

Recommendations

• Approximation by tree tensor networks in high dimensions: Sobolev and compositional functions
• TT ranks of approximate tensorizations of some smooth functions
• A continuous analogue of the tensor-train decomposition
• Approximation spaces of deep neural networks
• Higher-order principal component analysis for the approximation of tensors in tree-based low-rank formats

zbMATH Keywords

Besov spacesneural networkstensor networksapproximation spacesmatrix product statestensor trainsdirect (Jackson) and inverse (Bernstein) inequalities

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05) Nontrigonometric harmonic analysis involving wavelets and other special systems (42C40) Approximation by polynomials (41A10) Spline approximation (41A15) Abstract approximation theory (approximation in normed linear spaces and other abstract spaces) (41A65) Numerical approximation and computational geometry (primarily algorithms) (65D99)

Cites Work

• Deep learning
• Approximation of matrices with logarithmic number of parameters
• Hierarchical Singular Value Decomposition of Tensors
• Breaking the Curse of Dimensionality, Or How to Use SVD in Many Dimensions
• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• Theory and practice of finite elements.
• Title not available (Why is that?)
• Tensor spaces and numerical tensor calculus
• Inequalities on the Lambert \(W\) function and hyperpower function
• Interpolation of Besov Spaces
• An area law for one-dimensional quantum systems
• Optimal nonlinear approximation
• A practical introduction to tensor networks: Matrix product states and projected entangled pair states
• A new scheme for the tensor representation
• \(O(d \log N)\)-quantics approximation of \(N\)-\(d\) tensors in high-dimensional numerical modeling
• Adaptive near-optimal rank tensor approximation for high-dimensional operator equations
• Constructive representation of functions in low-rank tensor formats
• On manifolds of tensors of fixed TT-rank
• On minimal subspaces in tensor representations
• The Hackbusch conjecture on tensor formats
• Quantized tensor-structured finite elements for second-order elliptic PDEs in two dimensions
• Tree-based tensor formats
• Approximation rates for the hierarchical tensor format in periodic Sobolev spaces
• Tensor networks and hierarchical tensors for the solution of high-dimensional partial differential equations
• Analysis of tensor approximation schemes for continuous functions
• Causal structure of the entanglement renormalization ansatz
• Error bounds for approximations with deep ReLU networks
• Solving the quantum many-body problem with artificial neural networks
• Tensor-based multiscale method for diffusion problems in quasi-periodic heterogeneous media
• Real analysis. Theory of measure and integration
• On the Dirac-Frenkel variational principle on tensor Banach spaces
• Tensor networks for dimensionality reduction and large-scale optimization. I: Low-rank tensor decompositions
• Higher-order principal component analysis for the approximation of tensors in tree-based low-rank formats
• Approximation spaces of deep neural networks
• Deep ReLU networks and high-order finite element methods
• Tensor Networks for Dimensionality Reduction and Large-scale Optimization: Part 2 Applications and Future Perspectives
• Learning with tree tensor networks: complexity estimates and model selection
• Exponential ReLU DNN expression of holomorphic maps in high dimension
• The Hackbusch conjecture on tensor formats. II.
• Geometry of tree-based tensor formats in tensor Banach spaces
• Approximation of smoothness classes by deep rectifier networks
• LEARNING HIGH-DIMENSIONAL PROBABILITY DISTRIBUTIONS USING TREE TENSOR NETWORKS
• Approximation by tree tensor networks in high dimensions: Sobolev and compositional functions

Cited In (5)

• Low-rank tensor methods for partial differential equations
• Approximation by tree tensor networks in high dimensions: Sobolev and compositional functions
• Active Learning of Tree Tensor Networks using Optimal Least Squares
• RTNI—A symbolic integrator for Haar-random tensor networks
• Geometry of tree-based tensor formats in tensor Banach spaces

This page was built for publication: Approximation theory of tree tensor networks: tensorized univariate functions

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6076973)