Constructing a bpe tokenization DFA

DOI10.1016/J.TCS.2026.116131MaRDI QIDQ7319276zbMATH OpenFDO

Authors Martin Berglund, Willeke Martens, Brink van der Merwe

Publication date 25 August 2026

Published in Theoretical Computer Science (Search for Journal in Brave)

Full work available at URL https://doi.org/10.1016/J.TCS.2026.116131

Mathematics Subject Classification ID

Theory of computing (68Qxx)

Cites work

• Codes circulaires, automates locaux et entropie
• Constructing a BPE tokenization DFA
• Determinization of transducers over finite and infinite words.
• Families of locally testable languages
• Formalizing BPE tokenization
• Local languages and the Berry-Sethi algorithm

This page was built for publication: Constructing a bpe tokenization DFA

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q7319276)