piecemaker

Tools for Preparing Text for Tokenizers

Last update: 2 June 2023

Copyright license: Apache License

Software version identifier: 1.0.1, 1.0.0, 1.0.2

Tokenizers break text into pieces that are more usable by machine learning models. Many tokenizers share some preparation steps. This package provides those shared steps, along with a simple tokenizer.