piecemaker

Last update 2 June 2023

Software version identifier 1.0.1, 1.0.0, 1.0.2

Tokenizers break text into pieces that are more usable by machine learning models. Many tokenizers share some preparation steps. This package provides those shared steps, along with a simple tokenizer.

Cited in

(2)

This page was built for software: piecemaker