Universal regular conditional distributions via probabilistic transformers

DOI10.1007/S00365-023-09635-3zbMATH Open1529.68277arXiv2105.07743OpenAlexW4361018823MaRDI QIDQ6101232FDOQ6101232

Authors: Anastasis Kratsios 

Publication date: 20 June 2023

Published in: Constructive Approximation (Search for Journal in Brave)

Abstract: We introduce a deep learning model that can universally approximate regular conditional distributions (RCDs). The proposed model operates in three phases: first, it linearizes inputs from a given metric space $mathcalX$ to $mathbb{R}^d$ via a feature map, then a deep feedforward neural network processes these linearized features, and then the network's outputs are then transformed to the $1$-Wasserstein space $mathcal{P}_1(mathbb{R}^D)$ via a probabilistic extension of the attention mechanism of Bahdanau et al. (2014). Our model, called the extit{probabilistic transformer (PT)}, can approximate any continuous function from $mathbb{R}^d$ to $mathcal{P}_1(mathbb{R}^D)$ uniformly on compact sets, quantitatively. We identify two ways in which the PT avoids the curse of dimensionality when approximating $mathcal{P}_1(mathbb{R}^D)$-valued functions. The first strategy builds functions in $C(mathbb{R}^d,mathcal{P}_1(mathbb{R}^D))$ which can be efficiently approximated by a PT, uniformly on any given compact subset of $mathbb{R}^d$. In the second approach, given any function $f$ in $C(mathbb{R}^d,mathcal{P}_1(mathbb{R}^D))$, we build compact subsets of $mathbb{R}^d$ whereon $f$ can be efficiently approximated by a PT.

Full work available at URL: https://arxiv.org/abs/2105.07743

zbMATH Keywords

computational optimal transporttransformersuniversal approximationgeometric deep learningregular conditional distributionsmeasure-valued neural networks

Mathematics Subject Classification ID

Probability distributions: general theory (60E05) Optimal transportation (49Q22) Artificial neural networks and deep learning (68T07) Approximation by other special function classes (41A30) Integration and disintegration of measures (28A50)

Cites Work

• Title not available (Why is that?)
• Autoregressive Conditional Heteroscedasticity with Estimates of the Variance of United Kingdom Inflation
• Colloquium lectures on geometric measure theory
• Universal approximation bounds for superpositions of a sigmoidal function
• Regularization and Variable Selection Via the Elastic Net
• Ridge Regression: Biased Estimation for Nonorthogonal Problems
• Convex analysis and monotone operator theory in Hilbert spaces
• Title not available (Why is that?)
• An Invitation to Statistics in Wasserstein Space
• Title not available (Why is that?)
• Understanding Machine Learning
• Stochastic gradient boosting.
• Better Bootstrap Confidence Intervals
• Lectures on analysis on metric spaces
• Title not available (Why is that?)
• Title not available (Why is that?)
• Filling Riemannian manifolds
• Assouad's theorem with dimension independent of the snowflaking
• A note on generalized inverses
• Approximation by finitely supported measures
• Title not available (Why is that?)
• Lipschitz-free Banach spaces
• A logical calculus of the ideas immanent in nervous activity
• On embedding uniform and topological spaces
• Approximation by superpositions of a sigmoidal function
• An L¹ ergodic theorem with values in a non-positively curved space via a canonical barycenter map
• Probability theory. A comprehensive course
• Convergence of stochastic proximal gradient algorithm
• Title not available (Why is that?)
• Title not available (Why is that?)
• Approximation rates for neural networks with general activation functions
• Stochastic Calculus for Fractional Brownian Motion I. Theory
• Title not available (Why is that?)
• Locally flat imbeddings of topological manifolds
• Hardy spaces on Ahlfors-regular quasi metric spaces. A sharp theory
• On topological infinite deficiency
• Dimensions, embeddings, and attractors
• Bi-Lipschitz approximation by finite-dimensional imbeddings
• On the structure of Lipschitz-free spaces
• Metaphors in systolic geometry
• Introduction to Stochastic Analysis and Malliavin Calculus
• Universality of deep convolutional neural networks
• Title not available (Why is that?)
• Lipschitz Algebras
• Semmes surfaces and intrinsic Lipschitz graphs in the Heisenberg group
• A GEOMETRIC STUDY OF WASSERSTEIN SPACES: HADAMARD SPACES
• Title not available (Why is that?)
• Fractional Brownian motion and the Markov property
• Uniform decomposition of probability measures: quantization, clustering and rate of convergence
• Approximation spaces of deep neural networks
• Rejoinder: ``Nonparametric regression using deep neural networks with ReLU activation function
• The universal approximation property. Characterization, construction, representation, and existence
• Equivalence of approximation by convolutional neural networks and fully-connected networks
• Deep Network Approximation for Smooth Functions
• Optimal approximation rate of ReLU networks in terms of width and depth
• Deep Neural Networks, Generic Universal Interpolation, and Controlled ODEs
• Title not available (Why is that?)
• Quasisymmetric dimension distortion of Ahlfors regular subsets of a metric space
• Book Review: Optimal transport: old and new
• On the unique predual problem for Lipschitz spaces
• Title not available (Why is that?)
• Orders of continuity moduli of operators of almost the best approximation
• Title not available (Why is that?)

Cited In (1)

• Generative OrnsteinUhlenbeck markets via geometric deep learning

This page was built for publication: Universal regular conditional distributions via probabilistic transformers

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6101232)