Constructing compact BPE token DFAs

DOI10.1007/978-3-032-02602-6_3MaRDI QIDQ6861699zbMATH OpenFDO

Authors Martin Berglund, Anna Jonsson, Willeke Martens, Brink van der Merwe

Publication date 10 February 2026

Full work available at URL https://doi.org/10.1007/978-3-032-02602-6_3

Mathematics Subject Classification ID

Formal languages and automata (68Q45)

Cites work

• Compression of finite-state automata through failure transitions
• Constructing a BPE tokenization DFA
• Efficient string matching
• Formalizing BPE tokenization
• SOFSEM 2025: theory and practice of computer science. 50th international conference on current trends in theory and practice of computer science, SOFSEM 2025, Bratislava, Slovak Republic, January 20--23, 2025. Proceedings. Part I

This page was built for publication: Constructing compact BPE token DFAs

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6861699)