rtiktoken

Authors David Zimmermann, Roger Zurawicki, Authors of the dependent Rust crates

Last update 31 August 2026

Copyright license MIT license, File License

Software version identifier PACKAGES.rds, 0.0.6, 0.0.7, 0.11.0.2, 0.11.0.3

Description

A thin wrapper around the tiktoken-rs crate, allowing to encode text into Byte-Pair-Encoding (BPE) tokens and decode tokens back to text. This is useful to understand how Large Language Models (LLMs) perceive text.

This page was built for software: rtiktoken