tokenizers
From MaRDI portal
Software:28295
swMATH16425CRANtokenizersMaRDI QIDQ28295
Fast, Consistent Tokenization of Natural Language Text
Last update: 22 December 2022
Copyright license: MIT license, File License
Software version identifier: 0.3.0
Source code repository: https://github.com/cran/tokenizers