Universal regular conditional distributions via probabilistic transformers

DOI10.1007/S00365-023-09635-3MaRDI QIDQ6101232zbMATH OpenOpenAlexFDO

Authors Anastasis Kratsios

Publication date 20 June 2023

Published in Constructive Approximation (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2105.07743

zbMATH Keywords

computational optimal transporttransformersuniversal approximationgeometric deep learningregular conditional distributionsmeasure-valued neural networks

Mathematics Subject Classification ID

Probability distributions: general theory (60E05) Optimal transportation (49Q22) Artificial neural networks and deep learning (68T07) Approximation by other special function classes (41A30) Integration and disintegration of measures (28A50)

Abstract: We introduce a deep learning model that can universally approximate regular conditional distributions (RCDs). The proposed model operates in three phases: first, it linearizes inputs from a given metric space $mathcalX$ to $mathbb{R}^d$ via a feature map, then a deep feedforward neural network processes these linearized features, and then the network's outputs are then transformed to the $1$-Wasserstein space $mathcal{P}_1(mathbb{R}^D)$ via a probabilistic extension of the attention mechanism of Bahdanau et al. (2014). Our model, called the extit{probabilistic transformer (PT)}, can approximate any continuous function from $mathbb{R}^d$ to $mathcal{P}_1(mathbb{R}^D)$ uniformly on compact sets, quantitatively. We identify two ways in which the PT avoids the curse of dimensionality when approximating $mathcal{P}_1(mathbb{R}^D)$-valued functions. The first strategy builds functions in $C(mathbb{R}^d,mathcal{P}_1(mathbb{R}^D))$ which can be efficiently approximated by a PT, uniformly on any given compact subset of $mathbb{R}^d$. In the second approach, given any function $f$ in $C(mathbb{R}^d,mathcal{P}_1(mathbb{R}^D))$, we build compact subsets of $mathbb{R}^d$ whereon $f$ can be efficiently approximated by a PT.

Recommendations

• On the capacity of deep generative networks for approximating distributions
• Designing universal causal deep learning models: The geometric (Hyper)transformer
• High-dimensional distribution generation through deep neural networks
• Universal approximation on metric spaces
• Computing functions of random variables via reproducing kernel Hilbert space representations

Cites work

• scientific article; zbMATH DE number 1713116 (Why is no real title available?)
• scientific article; zbMATH DE number 6378127 (Why is no real title available?)
• scientific article; zbMATH DE number 444671 (Why is no real title available?)
• scientific article; zbMATH DE number 3423130 (Why is no real title available?)
• scientific article; zbMATH DE number 3133048 (Why is no real title available?)
• scientific article; zbMATH DE number 47995 (Why is no real title available?)
• scientific article; zbMATH DE number 3636876 (Why is no real title available?)
• scientific article; zbMATH DE number 1223845 (Why is no real title available?)
• scientific article; zbMATH DE number 1461253 (Why is no real title available?)
• scientific article; zbMATH DE number 1404748 (Why is no real title available?)
• scientific article; zbMATH DE number 1405266 (Why is no real title available?)
• scientific article; zbMATH DE number 7370609 (Why is no real title available?)
• scientific article; zbMATH DE number 7306919 (Why is no real title available?)
• A geometric study of Wasserstein spaces: Hadamard spaces
• A logical calculus of the ideas immanent in nervous activity
• A non-probabilistic proof of the Assouad embedding theorem with bounds on the dimension
• A note on generalized inverses
• An \(L^1\) ergodic theorem with values in a non-positively curved space via a canonical barycenter map
• An invitation to statistics in Wasserstein space
• Approximation by finitely supported measures
• Approximation by superpositions of a sigmoidal function
• Approximation rates for neural networks with general activation functions
• Approximation spaces of deep neural networks
• Assouad's theorem with dimension independent of the snowflaking
• Autoregressive Conditional Heteroscedasticity with Estimates of the Variance of United Kingdom Inflation
• Better Bootstrap Confidence Intervals
• Bi-Lipschitz approximation by finite-dimensional imbeddings
• Book Review: Optimal transport: old and new
• Colloquium lectures on geometric measure theory
• Convergence of stochastic proximal gradient algorithm
• Convex analysis and monotone operator theory in Hilbert spaces
• Deep Network Approximation for Smooth Functions
• Deep neural networks, generic universal interpolation, and controlled ODEs
• Dimensions, embeddings, and attractors
• Equivalence of approximation by convolutional neural networks and fully-connected networks
• Filling Riemannian manifolds
• Fractional Brownian motion and the Markov property
• Hardy spaces on Ahlfors-regular quasi metric spaces. A sharp theory
• Introduction to stochastic analysis and Malliavin calculus
• Lectures on analysis on metric spaces
• Lipschitz algebras
• Lipschitz-free Banach spaces
• Locally flat imbeddings of topological manifolds
• Metaphors in systolic geometry
• On embedding uniform and topological spaces
• On the structure of Lipschitz-free spaces
• On the unique predual problem for Lipschitz spaces
• On topological infinite deficiency
• Optimal approximation rate of ReLU networks in terms of width and depth
• Orders of continuity moduli of operators of almost the best approximation
• Probability theory. A comprehensive course
• Quasisymmetric dimension distortion of Ahlfors regular subsets of a metric space
• Regularization and Variable Selection Via the Elastic Net
• Rejoinder: ``Nonparametric regression using deep neural networks with ReLU activation function
• Ridge Regression: Biased Estimation for Nonorthogonal Problems
• Semmes surfaces and intrinsic Lipschitz graphs in the Heisenberg group
• Stochastic Calculus for Fractional Brownian Motion I. Theory
• Stochastic gradient boosting.
• The universal approximation property. Characterization, construction, representation, and existence
• Understanding machine learning. From theory to algorithms
• Uniform decomposition of probability measures: quantization, clustering and rate of convergence
• Universal approximation bounds for superpositions of a sigmoidal function
• Universality of deep convolutional neural networks

Cited in

(2)

• Generative OrnsteinUhlenbeck markets via geometric deep learning
• Designing universal causal deep learning models: The geometric (Hyper)transformer

This page was built for publication: Universal regular conditional distributions via probabilistic transformers

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6101232)