Skip to content
Open-Source Registry

cargo / tokenizers

tokenizers 1.0.0-rc.3

bitbison: clean, with notes, behavior clean, dependencies notes, posture not checked, pipeline clean, 1 of 1 jobs captured RisknoneVulnsnone
processes
411
operations
51K
packages
172
compiler runs
85
connections
1
job time
1m 20s
capture
756 KB
records
96K
bytes / record
7.8

Fastest implementation of the most used tokenizers in AI.

Build behaviourcompiled C/C++

38,237,111 downloads · Apache-2.0 · source · home page bpehuggingfacesimdtokenizerwordpiece View on crates.io

Artifact

file · retained · product · graph v2
1

Checks

13 of 14 clear Build-time dependency advisories
Build behavior✓✓✓✓✓✓✓✓
Supply chain✓▲✓✓✓
Pipeline✓

Detections

0

No detections

No abnormal activity detected across 51K flows.

Network

1 peers
  • static.crates.io:8080 the package manager · cargo (tokenizers) via crates.io · 25,327 files · went into an output ecosystem
Dependencies181Build behaviour
ahash0.8.12build script
aho-corasick1.1.5
alloca0.4.0build script
allocator-api20.2.21
anes0.1.6
anstyle1.0.14
arbitrary-chunks0.4.1
assert_approx_eq1.1.0
autocfg1.5.1
base640.13.1
base640.22.1
bitcannon0.1.0-rc.3
bitflags2.13.1
bitvec1.1.1
block-pseudorand0.1.2
bumpalo3.20.3
bytemuck1.25.2
byteorder1.5.0
cast0.3.0
castaway0.2.4
cc1.4.4
cfg-if1.0.4
chacha200.10.2
chiapos-chacha80.1.0
ciborium0.2.2
ciborium-io0.2.2
ciborium-ll0.2.2
clap4.6.6
clap_builder4.6.6
clap_lex1.1.0
colored3.1.1
compact_str0.9.1
console0.16.4
cpufeatures0.3.0
criterion0.5.1
criterion0.8.2
criterion-plot0.5.0
criterion-plot0.8.2
crossbeam-deque0.8.7build script
crossbeam-epoch0.9.20build script
crossbeam-utils0.8.22build script
crunchy0.2.4build script, not run
dary_heap0.3.9
either1.18.0
encode_unicode1.0.0
equivalent1.0.2
errno0.3.14
esaxx-rs0.1.10build script, not run
fastrand2.5.0
find-msvc-tools0.1.11
foldhash0.2.0
funty2.0.0
futures-core0.3.34
futures-task0.3.34
futures-util0.3.34
fxhash0.2.1
getrandom0.3.4build script
getrandom0.4.3build script
half2.7.1
hashbrown0.16.1
hermit-abi0.5.2
hifijson0.5.0
indicatif0.18.6
is-terminal0.4.17
itertools0.10.5
itertools0.13.0
itertools0.14.0
itertools0.15.0
itoa1.0.18
js-sys0.3.104
lazy_static1.5.0
ld-linux-x86-64.so.2system library
libc0.2.189build script
libc.so.6system library
libgcc_s.so.1system library
libm.so.6system library
libmvec.so.1system library
linux-raw-sys0.12.1
log0.4.33
mem_dbg0.4.4
mem_dbg-derive0.3.4proc macro
memchr2.8.3
minimal-lexical0.2.1
nanorand0.6.1
nom7.1.3
nu-ansi-term0.50.3
num-traits0.2.19build script
once_cell1.21.4
oorandom11.1.5
page_size0.6.0
partition0.1.2
pin-project-lite0.2.17
plotters0.3.7
plotters-backend0.3.7
plotters-svg0.3.7
portable-atomic1.15.0build script, not run
ppv-lite860.2.21
prefetch-index0.2.1
proc-macro21.0.107build script
ptr_hash2.1.2
quote1.0.47build script
r-efi5.3.0
r-efi6.0.0
radium0.7.0build script
rand0.10.2
rand_chacha0.10.0
rand_core0.10.1
rayon1.12.0
rayon-cond0.4.0
rayon-core1.13.0build script
rdst0.20.14
regex1.13.1
regex-automata0.4.18
regex-syntax0.8.11
rustix1.1.4build script
rustversion1.0.23build script, not runproc macro
ryu1.0.23
safe_arch1.2.0
same-file1.0.6
serde1.0.229build script
serde_core1.0.229build script
serde_derive1.0.229proc macro
serde_json1.0.151build script
sharded-slab0.1.7
shlex2.0.1
slab0.4.12
smallvec1.15.2
spm_precompiled0.1.4
static_assertions1.1.0
syn2.0.119
syn3.0.3
tap1.0.1
tempfile3.27.0
thiserror2.0.20build script
thiserror-impl2.0.20proc macro
thread_local1.1.10
tikv-jemalloc-sys0.5.4+5.3.0-patchedbuild script, not run
tikv-jemallocator0.5.4
tinytemplate1.2.1
tinyvec1.12.0
tinyvec_macros0.1.1
tk-convert0.1.0-rc.3
tk-encode0.1.0-rc.3
tk-serialize0.1.0-rc.3
tk-train0.1.0-rc.3
tracing0.1.44
tracing-attributes0.1.31proc macro
tracing-core0.1.36
tracing-log0.2.0
tracing-subscriber0.3.23
unicode-ident1.0.24
unicode-normalization0.1.25
unicode-segmentation1.13.3
unicode-width0.2.2
unicode_categories0.1.1
unit-prefix0.5.2
valuable0.1.1build script, not run
version_check0.9.5
voracious_radix_sort1.2.0
walkdir2.5.0
wasip21.0.4+wasi-0.2.12
wasm-bindgen0.2.127build script, not run
wasm-bindgen-macro0.2.127proc macro
wasm-bindgen-macro-support0.2.127
wasm-bindgen-shared0.2.127build script, not run
web-sys0.3.104
web-time1.1.0
wide1.6.1
winapi0.3.9build script, not run
winapi-i686-pc-windows-gnu0.4.0build script, not runprebuilt objects
winapi-util0.1.11
winapi-x86_64-pc-windows-gnu0.4.0build script, not runprebuilt objects
windows-link0.2.1
windows-sys0.61.2
wit-bindgen0.57.1build script, not runprebuilt objects
wyz0.5.1
xxhash-rust0.8.18
yada0.7.0
zerocopy0.8.56build script
zerocopy-derive0.8.56proc macro
zmij1.0.23build script
Versions43RiskVulnsPublished
1.0.0-rc.2not analyzed2026-09-21
0.23.2not analyzed2026-09-03
0.23.1not analyzed2026-04-27
0.22.2not analyzed2025-12-02
0.22.1not analyzed2025-09-19
0.22.0not analyzed2025-08-29
0.21.4not analyzed2025-07-28
0.21.4-dev.0not analyzed2025-07-04
0.21.2not analyzed2025-06-24
0.21.1not analyzed2025-03-13
0.21.0not analyzed2024-11-27
0.21.0-rc0not analyzed2024-11-27
0.20.4not analyzed2024-11-26
0.20.3not analyzed2024-11-05
0.20.2not analyzed2024-11-04
0.20.1not analyzed2024-10-10
0.20.0not analyzed2024-08-08
0.19.1not analyzed2024-04-17
0.19.0not analyzed2024-04-17
0.15.2not analyzed2024-02-12
0.15.1not analyzed2024-01-22
0.15.0not analyzed2023-11-14
0.14.1not analyzed2023-10-06
0.14.0not analyzed2023-09-07
0.14.0-rc.1not analyzed2023-09-07
0.13.4not analyzed2023-08-14
0.13.3not analyzed2023-04-05
0.13.2not analyzed2022-11-07
0.13.1not analyzed2022-10-06
0.13.0not analyzed2022-09-19
0.12.0yankednot analyzed2022-03-31
0.11.3not analyzed2022-02-28
0.11.0not analyzed2021-08-31
0.10.1not analyzed2020-04-09
0.10.0not analyzed2020-04-08
0.9.0not analyzed2020-03-30
0.8.0not analyzed2020-03-02
0.7.0not analyzed2020-02-05
0.6.0not analyzed2020-01-10
0.5.0not analyzed2019-10-09
0.2.0not analyzed2019-08-09
0.1.0not analyzed2019-08-08
1.0.0-rc.3nonenone

Raw data

capture
756 KB
records
96,362
bytes per record
7.8

Loading a sample of the capture

Want to see more? E-mail us at hello@bitbison.io about your use case.

1 of 1 jobs captured: build-test success · Generated 2026-10-19 · run 50f80452-8f00-42fa-aa4a-61e0387d9a8f

analysis 0.1.0 · 2026-10-19 · 10ce9f4 · graph v2