Space filling positionality and the Spiroformer

A transformer model is a kind of neural network architecture that is widely used for the analysis of sequential data. In this paper the authors consider how one might generalize transformer models to other geometric domains and deal with the absence of a well-defined global order. They propose a solution where the transition heads of a standard sequential transformer are replaced with transition heads that follow a space filling curve.\N\NThe authors create a transformer they call the Spiroformer, one that follows a polar spiral on the two-dimensional sphere.\N\NTheir model uses the transformer architecture to learn the dynamics of Hamiltonian vector fields on the sphere. The main idea is to transform a vector field reconstruction problem into a sequence-to-sequence learning task. The input is a sequence of vector field samples along the spiral, and the output is a prediction of the next vector field sample. The spiral trajectory, in traversing the two-sphere, connects the antipodal poles and guides positional encoding in order to capture the geometry and global connectivity.\N\NA data set was created to test the model. The authors found that the Spiroformer model learns dynamical features and can achieve high accuracy during training. Overfitting appears to be a concern, but the authors suggest several approaches to deal with this.\N\NRelated work is presented in [\textit{S. Cho} et al., Expert Syst. Appl. 238, Article ID 121853 (2024)].\N\NFor the entire collection see [Zbl 08117788].

Cites work

This page was built for publication: Space filling positionality and the Spiroformer

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6847918)