sentencepiece

CRANsentencepieceMaRDI QIDQ89656FDOQ89656

Text Tokenization using Byte Pair Encoding and Unigram Modelling

Jan Wijffels

Last update: 13 November 2022

Copyright license: Mozilla Public License, version 2.0

Software version identifier: 0.2.3

Cites work

• SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

This page was built for software: sentencepiece