Skip to content
Open-Source Registry

cargo / llm-tokenizer / 1.10.0

llm-tokenizer 1.10.0

bitbison: flagged, behavior flagged, dependencies notes, posture not checked, pipeline clean, 1 of 1 jobs captured RiskinfoVulnsnone
processes
662
operations
94K
packages
332
compiler runs
199
connections
3
job time
5m 23s
capture
1.0 MB
records
157K
bytes / record
6.6

Build behaviourcompiled C/C++

run e7c9050b-869f-460f-ae1d-ee3a6397d26d · built 2026-10-15 · complete View on crates.io

Graph

file · retained · product · graph v2
3

Checks

12 of 14 clear Sensitive data access and transfer Build-time dependency advisories
Build behavior✓▲✓✓✓✓✓✓
Supply chain✓▲✓✓✓
Pipeline✓

Detections

2
  • Code this build ran used live credentials
    tokenizer_cache_correctness_test-bf26ac599b40a911 › huggingface.co:443, us.aws.cdn.hf.co:443
    info
  • Code this build ran used live credentials
    tokenizer_integration-4db8e72e10ac9d75 › huggingface.co:443, us.aws.cdn.hf.co:443
    info

Detection

infowitnessed

Code this build ran used live credentials

Credentials used by code the build ran

Subject
tokenizer_cache_correctness_test-bf26ac599b40a911
Destination
huggingface.co:443, us.aws.cdn.hf.co:443
Jobs
build-test
Rule
credential-use

Evidence

  1. Connected after reading a credential storecause

    tokenizer_cache_correctness_test-bf26ac599b40a911 connected to huggingface.co:443 after reading /home/runner/.cache/huggingface/token

    source /home/runner/.cache/huggingface/tokentarget huggingface.co:443

    by /home/runner/work/smg/smg/target/release/deps/tokenizer_cache_correctness_test-bf26ac599b40a911

  2. Connected after reading a credential storecause

    tokenizer_cache_correctness_test-bf26ac599b40a911 connected to us.aws.cdn.hf.co:443 after reading /home/runner/.cache/huggingface/token

    source /home/runner/.cache/huggingface/tokentarget us.aws.cdn.hf.co:443

    by /home/runner/work/smg/smg/target/release/deps/tokenizer_cache_correctness_test-bf26ac599b40a911

What came before it

Loading the graph

Processes

build-test

cargo›cargo›tokenizer_cache_correctness_test-bf26ac599b40a911

Ran
/home/runner/work/smg/smg/target/release/deps/tokenizer_cache_correctness_test-bf26ac599b40a911
Invoked by
cargo test --release --locked --no-fail-fast
Lineage · 3 processes
  1. cargocargo test --release --locked --no-fail-fast5967
  2. cargo/home/runner/.rustup/toolchains/1.99.0-x86_64-unknown-linux-gnu/bin/cargo test --release --locked --no-fail-fast5967
  3. tokenizer_cache_correctness_test-bf26ac599b40a911/home/runner/work/smg/smg/target/release/deps/tokenizer_cache_correctness_test-bf26ac599b40a9119153

Remediation

A program this build made read a credential store and connected to the service it is for. Tests that need the runner's credentials make real calls with them; run them on a runner whose credentials are scoped to that, or record the use as accepted once it is reviewed.

Detection

infowitnessed

Code this build ran used live credentials

Credentials used by code the build ran

Subject
tokenizer_integration-4db8e72e10ac9d75
Destination
huggingface.co:443, us.aws.cdn.hf.co:443
Jobs
build-test
Rule
credential-use · smg

Evidence

  1. Connected after reading a credential storecause

    tokenizer_integration-4db8e72e10ac9d75 connected to huggingface.co:443 after reading /home/runner/.cache/huggingface/token

    source /home/runner/.cache/huggingface/tokentarget huggingface.co:443

    by /home/runner/work/smg/smg/target/release/deps/tokenizer_integration-4db8e72e10ac9d75

  2. Connected after reading a credential storecause

    tokenizer_integration-4db8e72e10ac9d75 connected to us.aws.cdn.hf.co:443 after reading /home/runner/.cache/huggingface/token

    source /home/runner/.cache/huggingface/tokentarget us.aws.cdn.hf.co:443

    by /home/runner/work/smg/smg/target/release/deps/tokenizer_integration-4db8e72e10ac9d75

What came before it

Loading the graph

Processes

build-test

cargo›cargo›tokenizer_integration-4db8e72e10ac9d75

Ran
/home/runner/work/smg/smg/target/release/deps/tokenizer_integration-4db8e72e10ac9d75
Invoked by
cargo test --release --locked --no-fail-fast
Lineage · 3 processes
  1. cargocargo test --release --locked --no-fail-fast5967
  2. cargo/home/runner/.rustup/toolchains/1.99.0-x86_64-unknown-linux-gnu/bin/cargo test --release --locked --no-fail-fast5967
  3. tokenizer_integration-4db8e72e10ac9d75/home/runner/work/smg/smg/target/release/deps/tokenizer_integration-4db8e72e10ac9d759154

Remediation

A program this build made read a credential store and connected to the service it is for. Tests that need the runner's credentials make real calls with them; run them on a runner whose credentials are scoped to that, or record the use as accepted once it is reviewed.

Network

3 peers
  • huggingface.co:443 llm-tokenizer's own build code · deepseek_v41_renderer-8c4172395b639cab (smg); deepseek_v4_renderer-17f4c013e0a28f5b (smg); kimi_k3_renderer-4ebf8b5ba010fa8f (smg); minimax_root_role_renderer-a209937a6361f9bc (smg); tokenizer_cache_correctness_test-bf26ac599b40a911; tokenizer_integration-4db8e72e10ac9d75 (smg) · Amazon.com, Inc. unjudged
  • us.aws.cdn.hf.co:443 llm-tokenizer's own build code · kimi_k3_renderer-4ebf8b5ba010fa8f (smg); tokenizer_cache_correctness_test-bf26ac599b40a911; tokenizer_integration-4db8e72e10ac9d75 (smg) · Amazon.com, Inc. unjudged
  • static.crates.io:8080 the package manager · cargo (smg) via crates.io · 22,213 files · went into an output ecosystem
Dependencies365Build behaviour
ahash0.8.12build script
aho-corasick1.1.5
alloca0.4.0build script
android_system_properties0.1.6
anes0.1.6
anstyle1.0.14
anyhow1.0.104build script
arrayvec0.7.8
atomic-waker1.1.2
autocfg1.5.1
base640.13.1
base640.22.1
base640.23.1
bit-set0.8.0
bit-vec0.8.0
bitflags1.3.2
bitflags2.13.2
blake31.8.7build script
bs580.5.1
bstr1.13.1
bumpalo3.20.3
bytes1.12.1
cast0.3.0
castaway0.2.4
cc1.5.1
cfg-if1.0.4
cfg_aliases0.2.2
chacha200.10.1
chrono0.4.45
ciborium0.2.2
ciborium-io0.2.2
ciborium-ll0.2.2
clap4.6.6
clap_builder4.6.6
clap_lex1.1.0
combine4.6.7
compact_str0.9.1
console0.16.4
constant_time_eq0.4.2
core-foundation0.10.1
core-foundation0.9.4
core-foundation-sys0.8.7
cpufeatures0.3.0
criterion0.8.2
criterion-plot0.8.2
crossbeam-deque0.8.7build script
crossbeam-epoch0.9.20build script
crossbeam-utils0.8.22build script
crunchy0.2.4build script, not run
daachorse3.0.3
darling0.20.11
darling0.23.0
darling0.24.1
darling_core0.20.11
darling_core0.23.0
darling_core0.24.1
darling_macro0.20.11proc macro
darling_macro0.23.0proc macro
darling_macro0.24.1proc macro
dary_heap0.3.9
dashmap6.2.1
defmt1.1.1build script, not run
defmt-macros1.1.1build script, not runproc macro
defmt-parser1.0.0
deranged0.5.8
derive_builder0.20.2
derive_builder_core0.20.2
derive_builder_macro0.20.2proc macro
dirs6.0.0
dirs-sys0.5.0
displaydoc0.2.7proc macro
dyn-clone1.0.20
either1.18.0
encode_unicode1.0.0
encoding_rs0.8.35
equivalent1.0.2
errno0.3.14
esaxx-rs0.1.10build script
fancy-regex0.19.2
fastrand2.5.0
find-msvc-tools0.1.14
fnv1.0.7
form_urlencoded1.2.2
futures0.3.34
futures-channel0.3.34
futures-core0.3.34
futures-executor0.3.34
futures-io0.3.34
futures-macro0.3.34proc macro
futures-sink0.3.34
futures-task0.3.34
futures-util0.3.34
getrandom0.2.17
getrandom0.3.4build script
getrandom0.4.3build script
h20.4.19
half2.7.1
hashbrown0.12.3
hashbrown0.14.5
hashbrown0.17.1
hermit-abi0.5.2
hex0.4.3
hf-hub0.5.0
http1.5.0
http-body1.1.0
http-body-util0.1.5
httparse1.10.1build script
hyper1.11.1
hyper-rustls0.27.9
hyper-util0.1.20
iana-time-zone0.1.65
iana-time-zone-haiku0.1.2build script, not run
icu_collections2.3.0
icu_locale_core2.3.0
icu_normalizer2.3.0
icu_normalizer_data2.3.0build script
icu_properties2.3.0
icu_properties_data2.3.0build script
icu_provider2.3.1
ident_case1.0.1
idna1.1.0
idna_adapter1.2.2
indexmap1.9.3build script, not run
indexmap2.14.2
indicatif0.18.6
ipnet2.12.1
itertools0.13.0
itertools0.14.0
itoa1.0.18
jiff0.2.35
jiff-core0.1.0
jiff-static0.2.35proc macro
jiff-tzdb0.1.8
jiff-tzdb-platform0.1.3
jni0.22.4build script, not run
jni-macros0.22.4build script, not runproc macro
jni-sys0.4.1
jni-sys-macros0.4.1proc macro
js-sys0.3.104
ld-linux-x86-64.so.2system library
libc0.2.190build script
libc.so.6system library
libgcc_s.so.1system library
libm.so.6system library
libmvec.so.1system library
libredox0.1.20
linux-raw-sys0.12.1
litemap0.8.3
lock_api0.4.14
log0.4.33
lru-slab0.1.2
macro_rules_attribute0.2.3
macro_rules_attribute-proc_macro0.2.3proc macro
memchr2.8.3
memo-map0.3.3
mime0.3.17
minijinja2.24.0
minijinja-contrib2.24.0
minimal-lexical0.2.1
mio1.2.2
monostate0.1.18
monostate-impl0.1.18proc macro
nom7.1.3
num-conv0.2.2
num-traits0.2.19build script
num_cpus1.17.0
once_cell1.21.4
onig6.5.3
onig_sys69.9.3build script
oorandom11.1.5
openai-protocol1.16.0
openssl-probe0.2.1
option-ext0.2.0
page_size0.6.0
parking_lot_core0.9.12build script
paste1.0.15build scriptproc macro
pastey0.2.3proc macro
percent-encoding2.3.2
pin-project-lite0.2.17
pkg-config0.3.34
plotters0.3.7
plotters-backend0.3.7
plotters-svg0.3.7
portable-atomic1.15.0build script
portable-atomic-util0.2.7build script, not run
potential_utf0.1.6
powerfmt0.2.0
ppv-lite860.2.21
proc-macro-error-attr33.1.0proc macro
proc-macro-error33.1.0
proc-macro21.0.107build script
quinn0.11.11build script, not run
quinn-proto0.11.18
quinn-udp0.5.15build script, not run
quote1.0.47build script
r-efi5.3.0
r-efi6.0.0
rand0.10.2
rand0.9.5
rand_chacha0.9.0
rand_core0.10.1
rand_core0.9.5
rand_pcg0.10.2
rayon1.12.0
rayon-cond0.4.0
rayon-core1.13.0build script
redox_syscall0.5.18
redox_users0.5.2
ref-cast1.0.27build script
ref-cast-impl1.0.27proc macro
regex1.13.1
regex-automata0.4.18
regex-syntax0.8.11
reqwest0.12.28
reqwest0.13.4
ring0.17.14build scriptprebuilt objects
rustc-hash2.1.3
rustc_version0.4.1
rustix1.1.4build script
rustls0.23.46build script
rustls-native-certs0.8.4
rustls-pki-types1.15.1
rustls-platform-verifier0.7.0
rustls-platform-verifier-android0.1.1
rustls-webpki0.103.14
rustversion1.0.23build scriptproc macro
ryu1.0.23
same-file1.0.6
schannel0.1.29
schemars0.9.0
schemars1.2.2
schemars_derive1.2.2proc macro
scopeguard1.2.0
security-framework3.7.0
security-framework-sys2.17.0
semver1.0.28
serde1.0.229build script
serde_core1.0.229build script
serde_derive1.0.229proc macro
serde_derive_internals0.30.0
serde_json1.0.151build script
serde_spanned1.1.1
serde_urlencoded0.7.1
serde_with3.24.0
serde_with_macros3.24.0proc macro
shlex2.0.1
simd_cesu81.2.0
simdutf80.1.5
slab0.4.12
smallvec1.15.2
socket20.6.5
spm_precompiled0.1.4
stable_deref_trait1.2.1
static_assertions1.1.0
strsim0.11.1
subtle2.6.1
syn2.0.119
syn3.0.3
sync_wrapper1.0.2
synstructure0.13.2
system-configuration0.7.0
system-configuration-sys0.6.0build script, not run
tempfile3.27.0
thiserror2.0.21build script
thiserror-impl2.0.21proc macro
thread_local1.1.10
tiktoken-rs0.12.1
time0.3.55
time-core0.1.9
time-macros0.2.32proc macro
tinystr0.8.4
tinytemplate1.2.1
tinyvec1.12.0
tinyvec_macros0.1.1
tokenizers0.23.2
tokio1.53.1
tokio-macros2.7.2proc macro
tokio-rustls0.26.5
tokio-util0.7.20
toml1.1.6+spec-1.1.0
toml_datetime1.1.1+spec-1.1.0
toml_parser1.1.3+spec-1.1.0
toml_writer1.1.2+spec-1.1.0
tower0.5.3
tower-http0.6.11
tower-layer0.3.3
tower-service0.3.3
tracing0.1.44
tracing-attributes0.1.31proc macro
tracing-core0.1.36
try-lock0.2.5
unicode-ident1.0.24
unicode-normalization0.1.25
unicode-normalization-alignments0.1.12
unicode-segmentation1.13.3
unicode-width0.2.2
unicode_categories0.1.1
unit-prefix0.5.2
untrusted0.9.0
url2.5.8
utf8_iter1.0.4
uuid1.28.0
validator0.20.0
validator_derive0.20.1proc macro
version_check0.9.5
walkdir2.5.0
want0.3.1
wasi0.11.1+wasi-snapshot-preview1
wasip21.0.4+wasi-0.2.12
wasm-bindgen0.2.127build script, not run
wasm-bindgen-futures0.4.77
wasm-bindgen-macro0.2.127proc macro
wasm-bindgen-macro-support0.2.127
wasm-bindgen-shared0.2.127build script, not run
wasm-streams0.4.2
web-sys0.3.104
web-time1.1.0
webpki-root-certs1.0.9
webpki-roots1.0.9
winapi0.3.9build script, not run
winapi-i686-pc-windows-gnu0.4.0build script, not runprebuilt objects
winapi-util0.1.11
winapi-x86_64-pc-windows-gnu0.4.0build script, not runprebuilt objects
windows-core0.62.2
windows-implement0.60.2proc macro
windows-interface0.59.3proc macro
windows-link0.2.1
windows-registry0.6.1
windows-result0.4.1
windows-strings0.5.1
windows-sys0.52.0
windows-sys0.60.2
windows-sys0.61.2
windows-targets0.52.6
windows-targets0.53.5
windows_aarch64_gnullvm0.52.6build script, not runprebuilt objects
windows_aarch64_gnullvm0.53.1build script, not runprebuilt objects
windows_aarch64_msvc0.52.6build script, not runprebuilt objects
windows_aarch64_msvc0.53.1build script, not runprebuilt objects
windows_i686_gnu0.52.6build script, not runprebuilt objects
windows_i686_gnu0.53.1build script, not runprebuilt objects
windows_i686_gnullvm0.52.6build script, not runprebuilt objects
windows_i686_gnullvm0.53.1build script, not runprebuilt objects
windows_i686_msvc0.52.6build script, not runprebuilt objects
windows_i686_msvc0.53.1build script, not runprebuilt objects
windows_x86_64_gnu0.52.6build script, not runprebuilt objects
windows_x86_64_gnu0.53.1build script, not runprebuilt objects
windows_x86_64_gnullvm0.52.6build script, not runprebuilt objects
windows_x86_64_gnullvm0.53.1build script, not runprebuilt objects
windows_x86_64_msvc0.52.6build script, not runprebuilt objects
windows_x86_64_msvc0.53.1build script, not runprebuilt objects
winnow1.0.4
wit-bindgen0.57.1build script, not runprebuilt objects
writeable0.6.4
yoke0.8.3
yoke-derive0.8.2proc macro
zerocopy0.8.56build script
zerocopy-derive0.8.56proc macro
zerofrom0.1.8
zerofrom-derive0.1.7proc macro
zeroize1.9.0
zerotrie0.2.5
zerovec0.11.8
zerovec-derive0.11.6proc macro
zmij1.0.23build script
Versions16RiskVulnsPublished
1.9.09d ago
1.8.0not analyzed2026-09-24
1.7.0not analyzed2026-08-25
1.6.0not analyzed2026-07-30
1.5.0not analyzed2026-07-18
1.4.2not analyzed2026-06-29
1.4.1not analyzed2026-06-21
1.4.0not analyzed2026-06-12
1.3.2not analyzed2026-04-09
1.3.1not analyzed2026-04-02
1.3.0not analyzed2026-03-10
1.2.0not analyzed2026-02-23
1.1.0not analyzed2026-02-13
1.0.0not analyzed2026-01-21
0.0.1not analyzed2026-01-06
1.10.0infonone

Raw data

capture
1.0 MB
records
156,579
bytes per record
6.6

Loading a sample of the capture

Want to see more? E-mail us at hello@bitbison.io about your use case.

1 of 1 jobs captured: build-test success · Generated 2026-10-15 · run e7c9050b-869f-460f-ae1d-ee3a6397d26d

analysis 0.1.0 · 2026-10-15 · 244c7b2 · graph v2