Embeddings between Barron spaces with higher order activation functions

Authors Tjeerd Jan Heeringa, Len Spek, Felix L. Schwenninger, Christoph Brune

Publication date 25 May 2023

Banach spaces of continuous, differentiable or analytic functions (46E15) Sobolev spaces and other spaces of ``smooth functions, embedding theorems, trace theorems (46E35) Measures and integration on abstract linear spaces (46G12)

Abstract: The approximation properties of infinitely wide shallow neural networks heavily depend on the choice of the activation function. To understand this influence, we study embeddings between Barron spaces with different activation functions. These embeddings are proven by providing push-forward maps on the measures

m u

used to represent functions

f

. An activation function of particular interest is the rectified power unit (

o p e r a t o r n a m e R e P U

) given by

o p e r a t o r n a m e {R e P U}_{s} (x) = m a x (0, x)^{s}

. For many commonly used activation functions, the well-known Taylor remainder theorem can be used to construct a push-forward map, which allows us to prove the embedding of the associated Barron space into a Barron space with a

o p e r a t o r n a m e R e P U

as activation function. Moreover, the Barron spaces associated with the

o p e r a t o r n a m e {R e P U}_{s}

have a hierarchical structure similar to the Sobolev spaces

H^{m}

.

This page was built for publication: Embeddings between Barron spaces with higher order activation functions

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6437936)