Constructing a BPE tokenization DFA

DOI10.1007/978-3-031-71112-1_5MaRDI QIDQ6666787zbMATH OpenFDO

Authors Martin Berglund, Willeke Martens, Brink van der Merwe

Publication date 20 January 2025

Full work available at URL https://doi.org/10.1007/978-3-031-71112-1_5

Mathematics Subject Classification ID

Formal languages and automata (68Q45)

Cited in

(2)

• Constructing compact BPE token DFAs
• Constructing a bpe tokenization DFA

This page was built for publication: Constructing a BPE tokenization DFA

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6666787)