piecemaker (Q111109): Difference between revisions
From MaRDI portal
Changed an Item |
Changed an Item |
||
| Property / MaRDI profile type | |||
| Property / MaRDI profile type: MaRDI software profile / rank | |||
Normal rank | |||
Revision as of 13:32, 7 March 2024
Tools for Preparing Text for Tokenizers
| Language | Label | Description | Also known as |
|---|---|---|---|
| English | piecemaker |
Tools for Preparing Text for Tokenizers |
Statements
2 June 2023
0 references
Tokenizers break text into pieces that are more usable by machine learning models. Many tokenizers share some preparation steps. This package provides those shared steps, along with a simple tokenizer.
0 references
