Skip to content
Open-Source Registry

Search 2 packages

found
reviewed
build behaviour
Packages1–2 of 2VersionRiskVulnsDescriptionDownloadsWhen
bitcannon cargo0.1.0-rc.3nonenoneSIMD Unicode atom classification + bitstream pre-tokenization: one classify pass into byte-exact token spans (cl100k, o200k, GPT-2/ByteLevel, DeepSeek,…2.1K1d ago
magi-tokenizer cargo0.0.1nonenoneExact, fast o200k_base BPE tokenizer with build-time tables and a hand-written pretokenizer172d ago