Search 2 packages
| Packages1–2 of 2 | Version | Risk | Vulns | Description | Downloads | When |
|---|---|---|---|---|---|---|
| bitcannon cargo | 0.1.0-rc.3 | none | none | SIMD Unicode atom classification + bitstream pre-tokenization: one classify pass into byte-exact token spans (cl100k, o200k, GPT-2/ByteLevel, DeepSeek,… | 2.1K | 1d ago |
| magi-tokenizer cargo | 0.0.1 | none | none | Exact, fast o200k_base BPE tokenizer with build-time tables and a hand-written pretokenizer | 17 | 2d ago |
