Skip to content
Open-Source Registry

cargo / magi-tokenizer

magi-tokenizer 0.0.1

bitbison: clean, behavior clean, dependencies clean, posture not checked, pipeline clean, 1 of 1 jobs captured RisknoneVulnsnone
processes
167
operations
1.7K
packages
5
compiler runs
2
connections
0
job time
18s
capture
49 KB
records
3.9K
bytes / record
12.4

Exact, fast o200k_base BPE tokenizer with build-time tables and a hand-written pretokenizer

Build behaviourhas a build script

17 downloads · Apache-2.0 AND MIT AND Unicode-3.0 · published 2026-10-17 · source View on crates.io

Artifact

file · retained · product · graph v2
1

Checks

14 of 14 clear
Build behavior✓✓✓✓✓✓✓✓
Supply chain✓✓✓✓✓
Pipeline✓

Detections

0

No detections

No abnormal activity detected across 1.7K flows.

Dependencies5Build behaviour
ld-linux-x86-64.so.2system library
libc.so.6system library
libgcc_s.so.1system library
libm.so.6system library
libmvec.so.1system library
Versions1RiskVulnsPublished
0.0.1nonenone2d ago

Raw data

capture
49 KB
records
3,932
bytes per record
12.4

Loading a sample of the capture

Want to see more? E-mail us at hello@bitbison.io about your use case.

1 of 1 jobs captured: build-test success · Generated 2026-10-18 · run e7a74155-e98e-4542-a519-9d0c8fd20016

analysis 0.1.0 · 2026-10-17 · 0abbde9 · graph v2