Approximation theory of tree tensor networks: tensorized univariate functions

DOI10.1007/S00365-023-09620-WMaRDI QIDQ6076973zbMATH OpenFDO

Authors Mazen Ali, A. Nouy

Publication date 17 October 2023

Published in Constructive Approximation (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2007.00118

zbMATH Keywords

Besov spacesneural networkstensor networksapproximation spacesmatrix product statestensor trainsdirect (Jackson) and inverse (Bernstein) inequalities

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05) Nontrigonometric harmonic analysis involving wavelets and other special systems (42C40) Approximation by polynomials (41A10) Spline approximation (41A15) Abstract approximation theory (approximation in normed linear spaces and other abstract spaces) (41A65) Numerical approximation and computational geometry (primarily algorithms) (65D99)

Abstract: We study the approximation of functions by tensor networks (TNs). We show that Lebesgue $L^p$-spaces in one dimension can be identified with tensor product spaces of arbitrary order through tensorization. We use this tensor product structure to define subsets of $L^p$ of rank-structured functions of finite representation complexity. These subsets are then used to define different approximation classes of tensor networks, associated with different measures of complexity. These approximation classes are shown to be quasi-normed linear spaces. We study some elementary properties and relationships of said spaces. In part II of this work, we will show that classical smoothness (Besov) spaces are continuously embedded into these approximation classes. We will also show that functions in these approximation classes do not possess any Besov smoothness, unless one restricts the depth of the tensor networks. The results of this work are both an analysis of the approximation spaces of TNs and a study of the expressivity of a particular type of neural networks (NN) -- namely feed-forward sum-product networks with sparse architecture. The input variables of this network result from the tensorization step, interpreted as a particular featuring step which can also be implemented with a neural network with a specific architecture. We point out interesting parallels to recent results on the expressivity of rectified linear unit (ReLU) networks -- currently one of the most popular type of NNs.

Recommendations

• Approximation by tree tensor networks in high dimensions: Sobolev and compositional functions
• TT ranks of approximate tensorizations of some smooth functions
• A continuous analogue of the tensor-train decomposition
• Approximation spaces of deep neural networks
• Higher-order principal component analysis for the approximation of tensors in tree-based low-rank formats

Cites work

• scientific article; zbMATH DE number 192914 (Why is no real title available?)
• scientific article; zbMATH DE number 1215245 (Why is no real title available?)
• scientific article; zbMATH DE number 477682 (Why is no real title available?)
• scientific article; zbMATH DE number 1862742 (Why is no real title available?)
• A new scheme for the tensor representation
• A practical introduction to tensor networks: Matrix product states and projected entangled pair states
• Adaptive near-optimal rank tensor approximation for high-dimensional operator equations
• An area law for one-dimensional quantum systems
• Analysis of tensor approximation schemes for continuous functions
• Approximation by tree tensor networks in high dimensions: Sobolev and compositional functions
• Approximation of matrices with logarithmic number of parameters
• Approximation of smoothness classes by deep rectifier networks
• Approximation rates for the hierarchical tensor format in periodic Sobolev spaces
• Approximation spaces of deep neural networks
• Breaking the Curse of Dimensionality, Or How to Use SVD in Many Dimensions
• Causal structure of the entanglement renormalization ansatz
• Constructive representation of functions in low-rank tensor formats
• Deep ReLU networks and high-order finite element methods
• Deep learning
• Error bounds for approximations with deep ReLU networks
• Exponential ReLU DNN expression of holomorphic maps in high dimension
• Geometry of tree-based tensor formats in tensor Banach spaces
• Hierarchical Singular Value Decomposition of Tensors
• Higher-order principal component analysis for the approximation of tensors in tree-based low-rank formats
• Inequalities on the Lambert \(W\) function and hyperpower function
• Interpolation of Besov Spaces
• LEARNING HIGH-DIMENSIONAL PROBABILITY DISTRIBUTIONS USING TREE TENSOR NETWORKS
• Learning with tree tensor networks: complexity estimates and model selection
• On manifolds of tensors of fixed TT-rank
• On minimal subspaces in tensor representations
• On the Dirac-Frenkel variational principle on tensor Banach spaces
• Optimal nonlinear approximation
• Quantized tensor-structured finite elements for second-order elliptic PDEs in two dimensions
• Real analysis. Theory of measure and integration
• Solving the quantum many-body problem with artificial neural networks
• Tensor Networks for Dimensionality Reduction and Large-scale Optimization: Part 2 Applications and Future Perspectives
• Tensor networks and hierarchical tensors for the solution of high-dimensional partial differential equations
• Tensor networks for dimensionality reduction and large-scale optimization. I: Low-rank tensor decompositions
• Tensor spaces and numerical tensor calculus
• Tensor-based multiscale method for diffusion problems in quasi-periodic heterogeneous media
• The Hackbusch conjecture on tensor formats
• The Hackbusch conjecture on tensor formats. II.
• Theory and practice of finite elements.
• Tree-based tensor formats
• \(O(d \log N)\)-quantics approximation of \(N\)-\(d\) tensors in high-dimensional numerical modeling

Cited in

(5)

• Low-rank tensor methods for partial differential equations
• Approximation by tree tensor networks in high dimensions: Sobolev and compositional functions
• Active Learning of Tree Tensor Networks using Optimal Least Squares
• RTNI—A symbolic integrator for Haar-random tensor networks
• Geometry of tree-based tensor formats in tensor Banach spaces

This page was built for publication: Approximation theory of tree tensor networks: tensorized univariate functions

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6076973)