Skip to content
Open-Source Registry

cargo / tk-serialize

tk-serialize 0.1.0-rc.3

bitbison: clean, with notes, behavior clean, dependencies notes, posture not checked, pipeline clean, 1 of 1 jobs captured RisknoneVulnsnone
processes
414
operations
32K
packages
136
compiler runs
63
connections
1
job time
1m 07s
capture
447 KB
records
59K
bytes / record
7.6

The reader for the canonical `tokenizer.json` format: a `hifijson` parse and abuilder that constructs `tk-encode`'s runtime pipeline directly, with no serdeanywhere. Legacy config shapes are `tk-convert`'s job, not this crate's.

Build behaviourcompiled C/C++

1,903 downloads · Apache-2.0 · source · home page bpehuggingfacenlptokenizerwordpiece View on crates.io

Artifact

file · retained · product · graph v2
1

Checks

13 of 14 clear Build-time dependency advisories
Build behavior✓✓✓✓✓✓✓✓
Supply chain✓▲✓✓✓
Pipeline✓

Detections

0

No detections

No abnormal activity detected across 32K flows.

Network

1 peers
  • static.crates.io:8080 the package manager · cargo (tokenizers) via crates.io · 13,043 files · went into an output ecosystem
Dependencies143Build behaviour
ahash0.8.12build script
aho-corasick1.1.5
alloca0.4.0build script
allocator-api20.2.21
anes0.1.6
anstyle1.0.14
arbitrary-chunks0.4.1
autocfg1.5.1
base640.22.1
bitcannon0.1.0-rc.3
bitflags2.13.1
bitvec1.1.1
block-pseudorand0.1.2
bumpalo3.20.3
bytemuck1.25.2
byteorder1.5.0
cast0.3.0
cc1.4.4
cfg-if1.0.4
chacha200.10.2
chiapos-chacha80.1.0
ciborium0.2.2
ciborium-io0.2.2
ciborium-ll0.2.2
clap4.6.6
clap_builder4.6.6
clap_lex1.1.0
colored3.1.1
cpufeatures0.3.0
criterion0.5.1
criterion0.8.2
criterion-plot0.5.0
criterion-plot0.8.2
crossbeam-deque0.8.7build script
crossbeam-epoch0.9.20build script
crossbeam-utils0.8.22build script
crunchy0.2.4build script, not run
dary_heap0.3.9
either1.18.0
equivalent1.0.2
errno0.3.14
fastrand2.5.0
find-msvc-tools0.1.11
foldhash0.2.0
funty2.0.0
futures-core0.3.34
futures-task0.3.34
futures-util0.3.34
fxhash0.2.1
getrandom0.3.4build script
getrandom0.4.3build script
half2.7.1
hashbrown0.16.1
hermit-abi0.5.2
hifijson0.5.0
is-terminal0.4.17
itertools0.10.5
itertools0.13.0
itertools0.15.0
itoa1.0.18
js-sys0.3.104
ld-linux-x86-64.so.2system library
libc0.2.189build script
libc.so.6system library
libgcc_s.so.1system library
libm.so.6system library
libmvec.so.1system library
linux-raw-sys0.12.1
log0.4.33
mem_dbg0.4.4
mem_dbg-derive0.3.4proc macro
memchr2.8.3
nanorand0.6.1
num-traits0.2.19build script
once_cell1.21.4
oorandom11.1.5
page_size0.6.0
partition0.1.2
pin-project-lite0.2.17
plotters0.3.7
plotters-backend0.3.7
plotters-svg0.3.7
ppv-lite860.2.21
prefetch-index0.2.1
proc-macro21.0.107build script
ptr_hash2.1.2
quote1.0.47build script
r-efi5.3.0
r-efi6.0.0
radium0.7.0build script
rand0.10.2
rand_chacha0.10.0
rand_core0.10.1
rayon1.12.0
rayon-core1.13.0build script
rdst0.20.14
regex1.13.1
regex-automata0.4.18
regex-syntax0.8.11
rustix1.1.4build script
rustversion1.0.23build script, not runproc macro
safe_arch1.2.0
same-file1.0.6
serde1.0.229build script
serde_core1.0.229build script
serde_derive1.0.229proc macro
serde_json1.0.151build script
shlex2.0.1
slab0.4.12
syn2.0.119
syn3.0.3
tap1.0.1
tempfile3.27.0
thiserror2.0.20build script
thiserror-impl2.0.20proc macro
tikv-jemalloc-sys0.5.4+5.3.0-patchedbuild script, not run
tikv-jemallocator0.5.4
tinytemplate1.2.1
tk-encode0.1.0-rc.3
unicode-ident1.0.24
unicode_categories0.1.1
version_check0.9.5
voracious_radix_sort1.2.0
walkdir2.5.0
wasip21.0.4+wasi-0.2.12
wasm-bindgen0.2.127build script, not run
wasm-bindgen-macro0.2.127proc macro
wasm-bindgen-macro-support0.2.127
wasm-bindgen-shared0.2.127build script, not run
web-sys0.3.104
wide1.6.1
winapi0.3.9build script, not run
winapi-i686-pc-windows-gnu0.4.0build script, not runprebuilt objects
winapi-util0.1.11
winapi-x86_64-pc-windows-gnu0.4.0build script, not runprebuilt objects
windows-link0.2.1
windows-sys0.61.2
wit-bindgen0.57.1build script, not runprebuilt objects
wyz0.5.1
xxhash-rust0.8.18
zerocopy0.8.56build script
zerocopy-derive0.8.56proc macro
zmij1.0.23build script
Versions4RiskVulnsPublished
0.1.0-rc.2not analyzed2026-09-21
0.1.0-rc.1not analyzed2026-09-21
0.1.0-dev.0not analyzed2026-09-21
0.1.0-rc.3nonenone

Raw data

capture
447 KB
records
59,212
bytes per record
7.6

Loading a sample of the capture

Want to see more? E-mail us at hello@bitbison.io about your use case.

1 of 1 jobs captured: build-test success · Generated 2026-10-19 · run c916c356-8201-44c4-856e-6682c61715e0

analysis 0.1.0 · 2026-10-19 · 10ce9f4 · graph v2