piecemaker (Q111109): Difference between revisions
From MaRDI portal
Removed claim: author (P16): Jonathan Bratt (Q111027) |
Added link to MaRDI item. |
||||||||||||||
(7 intermediate revisions by one other user not shown) | |||||||||||||||
Property / last update | |||||||||||||||
| |||||||||||||||
Property / last update: 3 March 2022 / rank | |||||||||||||||
Property / copyright license | |||||||||||||||
Property / copyright license: Apache License / rank | |||||||||||||||
Property / copyright license: Apache License / qualifier | |||||||||||||||
Property / depends on software | |||||||||||||||
Property / depends on software: R / rank | |||||||||||||||
Property / imports | |||||||||||||||
Property / imports: rlang / rank | |||||||||||||||
Property / imports | |||||||||||||||
Property / imports: stringi / rank | |||||||||||||||
Property / imports | |||||||||||||||
Property / imports: stringr / rank | |||||||||||||||
Property / software version identifier | |||||||||||||||
1.0.0 | |||||||||||||||
Property / software version identifier: 1.0.0 / rank | |||||||||||||||
Normal rank | |||||||||||||||
Property / software version identifier: 1.0.0 / qualifier | |||||||||||||||
publication date: 6 August 2021
| |||||||||||||||
Property / software version identifier | |||||||||||||||
1.0.2 | |||||||||||||||
Property / software version identifier: 1.0.2 / rank | |||||||||||||||
Normal rank | |||||||||||||||
Property / software version identifier: 1.0.2 / qualifier | |||||||||||||||
publication date: 2 June 2023
| |||||||||||||||
Property / last update | |||||||||||||||
2 June 2023
| |||||||||||||||
Property / last update: 2 June 2023 / rank | |||||||||||||||
Normal rank | |||||||||||||||
Property / description | |||||||||||||||
Tokenizers break text into pieces that are more usable by machine learning models. Many tokenizers share some preparation steps. This package provides those shared steps, along with a simple tokenizer. | |||||||||||||||
Property / description: Tokenizers break text into pieces that are more usable by machine learning models. Many tokenizers share some preparation steps. This package provides those shared steps, along with a simple tokenizer. / rank | |||||||||||||||
Normal rank | |||||||||||||||
Property / author | |||||||||||||||
Property / author: Jon Harmon / rank | |||||||||||||||
Normal rank | |||||||||||||||
Property / author | |||||||||||||||
Property / author: Jonathan Bratt / rank | |||||||||||||||
Normal rank | |||||||||||||||
Property / copyright license | |||||||||||||||
Property / copyright license: Apache License / rank | |||||||||||||||
Normal rank | |||||||||||||||
Property / copyright license: Apache License / qualifier | |||||||||||||||
edition/version: ≥ 2 (English) | |||||||||||||||
Property / depends on software | |||||||||||||||
Property / depends on software: R / rank | |||||||||||||||
Normal rank | |||||||||||||||
Property / depends on software: R / qualifier | |||||||||||||||
software version identifier: ≥ 2.10 | |||||||||||||||
Property / imports | |||||||||||||||
Property / imports: cli / rank | |||||||||||||||
Normal rank | |||||||||||||||
Property / imports | |||||||||||||||
Property / imports: glue / rank | |||||||||||||||
Normal rank | |||||||||||||||
Property / imports | |||||||||||||||
Property / imports: rlang / rank | |||||||||||||||
Normal rank | |||||||||||||||
Property / imports: rlang / qualifier | |||||||||||||||
software version identifier: ≥ 0.4.2 | |||||||||||||||
Property / imports | |||||||||||||||
Property / imports: stringi / rank | |||||||||||||||
Normal rank | |||||||||||||||
Property / imports | |||||||||||||||
Property / imports: stringr / rank | |||||||||||||||
Normal rank | |||||||||||||||
Property / MaRDI profile type | |||||||||||||||
Property / MaRDI profile type: MaRDI software profile / rank | |||||||||||||||
Normal rank | |||||||||||||||
links / mardi / name | links / mardi / name | ||||||||||||||
Latest revision as of 18:56, 12 March 2024
Tools for Preparing Text for Tokenizers
Language | Label | Description | Also known as |
---|---|---|---|
English | piecemaker |
Tools for Preparing Text for Tokenizers |
Statements
2 June 2023
0 references
Tokenizers break text into pieces that are more usable by machine learning models. Many tokenizers share some preparation steps. This package provides those shared steps, along with a simple tokenizer.
0 references