Training-Free Universal Approximation by Prompting Random Transformers

DOI10.48550/arXiv.2608.09558MaRDI QIDQ7284795arXivFDO

Authors Rongjie Lai, Alexander Hsu

Publication date 10 August 2026

Published in arXiv

Tldr We show that pretraining is optional for transformers: a single-layer random softmax attention network can approximate any Hölder function on a compact manifold with an appropriate prompt, which we construct explicitly as solutions to linear systems matching attention logits to Gaussian kernel exponents under mild rank conditions on the weights.

Full work available at URL https://doi.org/10.48550/arXiv.2608.09558

arXiv classification

cs.LGmath.NAmath.ST

Cites work

• Approximation by superpositions of a sigmoidal function
• Attention Is All You Need
• Error bounds for approximations with deep ReLU networks
• Extreme learning machine: Theory and applications
• High-Dimensional Probability: An Introduction with Applications in Data Science
• In-Context Convergence of Transformers
• Local polynomial regression on unknown manifolds
• Local polynomial regression on unknown manifolds
• Prompt Engineering Through the Lens of Optimal Control
• Prompting a Pretrained Transformer Can Be a Universal Approximator
• Real-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations
• Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection
• Universal approximation bounds for superpositions of a sigmoidal function

This page was built for publication: Training-Free Universal Approximation by Prompting Random Transformers

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q7284795)