Skip to content
Open-Source Registry

cargo / rust_tokenizers

rust_tokenizers 8.1.1

processes
441
operations
48K
packages
224
compiler runs
45
connections
1
job time
1m 55s
capture
614 KB
records
87K
bytes / record
7.0

High performance tokenizers for Rust

clean

Build behaviourhas a build scriptcompiled C/C++

448,232 downloads · Apache-2.0 · published 2023-10-01 · source · cargo

machine-learning nlp tokenizer

Release 8.1.1 run bc10c9e9-0736-4d1e-94fa-79d1c90917b7 · built 2026-10-08

Artifact

file · retained · product
1
ChecksDetections
Build behavior
✓ Network activity—
✓ Sensitive data access and transfer—
✓ Cross-process access—
✓ Build file changes—
✓ Build script activity—
✓ Data flow analysis—
✓ Filesystem writes—
✓ Context collection—
Supply chain
✓ Known vulnerabilities—
▲ Build-time dependency advisories—
✓ Dependency advisories—
✓ Declared and observed dependencies—
✓ Dependency and toolchain versions—
Pipeline
✓ Pipeline coverage—

Detections

0

No detections

No abnormal activity detected across 48K flows.

Network

0 peers
    cargo fetched 17,293 files from static.crates.io

    Raw data

    capture
    614 KB
    records
    87,266
    bytes per record
    7.0

    Loading a sample of the capture

    Want to see more? E-mail us at hello@bitbison.io about your use case.

    Dependencies

    240 packages, 28 with a build script that ran, 18 doing more than compiling
    PackageVersionUsageBuild behaviour
    vcpkg0.2.15observedprebuilt objects
    winapi-i686-pc-windows-gnu0.4.0observedbuild script, not runprebuilt objects
    winapi-x86_64-pc-windows-gnu0.4.0observedbuild script, not runprebuilt objects
    windows_aarch64_gnullvm0.48.5observedbuild script, not runprebuilt objects
    windows_aarch64_gnullvm0.52.6observedbuild script, not runprebuilt objects
    windows_aarch64_msvc0.48.5observedbuild script, not runprebuilt objects
    windows_aarch64_msvc0.52.6observedbuild script, not runprebuilt objects
    windows_i686_gnu0.48.5observedbuild script, not runprebuilt objects
    windows_i686_gnu0.52.6observedbuild script, not runprebuilt objects
    windows_i686_gnullvm0.52.6observedbuild script, not runprebuilt objects
    windows_i686_msvc0.48.5observedbuild script, not runprebuilt objects
    windows_i686_msvc0.52.6observedbuild script, not runprebuilt objects
    windows_x86_64_gnu0.48.5observedbuild script, not runprebuilt objects
    windows_x86_64_gnu0.52.6observedbuild script, not runprebuilt objects
    windows_x86_64_gnullvm0.48.5observedbuild script, not runprebuilt objects
    windows_x86_64_gnullvm0.52.6observedbuild script, not runprebuilt objects
    windows_x86_64_msvc0.48.5observedbuild script, not runprebuilt objects
    windows_x86_64_msvc0.52.6observedbuild script, not runprebuilt objects
    ahash0.8.12observedbuild script
    anyhow1.0.104observedbuild script, not run
    bzip2-sys0.1.13+1.0.8observedbuild script
    crc32fast1.5.2observedbuild script
    crossbeam-deque0.8.8observedbuild script
    crossbeam-epoch0.9.21observedbuild script
    crossbeam-utils0.8.23observedbuild script
    displaydoc0.2.7observedproc macro
    encoding_rs0.8.42observedbuild script
    generic-array0.14.7observedbuild script
    getrandom0.4.3observedbuild script
    httparse1.10.1observedbuild script
    icu_normalizer_data2.3.0observedbuild script
    icu_properties_data2.3.0observedbuild script
    ld-linux-x86-64.so.2linkedsystem library
    libc0.2.190observedbuild script
    libc.so.6linkedsystem library
    libgcc_s.so.1linkedsystem library
    libm.so.6linkedsystem library
    libmvec.so.1linkedsystem library
    multiversion_no_op1.0.0observedproc macro
    native-tls0.2.18observedbuild script
    openssl0.10.81observedbuild script
    openssl-macros0.1.1observedproc macro
    openssl-sys0.9.117observedbuild script
    proc-macro21.0.107observedbuild script
    protobuf direct2.28.0observedbuild script
    quote1.0.47observedbuild script
    rayon-core1.13.0observedbuild script
    rustix1.1.5observedbuild script
    rustversion1.0.23observedbuild scriptproc macro
    serde direct1.0.229observedbuild script
    serde_core1.0.229observedbuild script
    serde_derive1.0.229observedproc macro
    serde_json direct1.0.151observedbuild script
    system-configuration-sys0.5.0observedbuild script, not run
    thiserror direct1.0.69observedbuild script
    thiserror-impl1.0.69observedproc macro
    wasm-bindgen0.2.129observedbuild script, not run
    wasm-bindgen-macro0.2.129observedproc macro
    wasm-bindgen-shared0.2.129observedbuild script, not run
    winapi0.3.9observedbuild script, not run
    yoke-derive0.8.4observedproc macro
    zerocopy0.8.62observedbuild script
    zerocopy-derive0.8.62observedproc macro
    zerofrom-derive0.1.8observedproc macro
    zerovec-derive0.11.6observedproc macro
    zmij1.0.23observedbuild script
    zstd-safe5.0.2+zstd.1.5.2observedbuild script, not run
    zstd-sys2.1.0+zstd.1.5.7observedbuild script, not run
    adler22.0.1observed
    aes0.8.4observed
    aho-corasick1.1.5observed
    allocator-api20.2.21observed
    base640.21.7observed
    base64ct1.8.3observed
    bitflags1.3.2observed
    bitflags2.13.2observed
    block-buffer0.10.4observed
    bumpalo3.20.3observed
    byteorder1.5.0observed
    bytes1.12.1observed
    bzip20.4.4observed
    cached-path0.6.2observed
    cc1.6.0observed
    cfg-if1.0.5observed
    cipher0.4.4observed
    console0.16.6observed
    constant_time_eq0.1.5observed
    core-foundation0.10.1observed
    core-foundation0.9.4observed
    core-foundation-sys0.8.7observed
    core_detect1.0.0observed
    cpufeatures0.2.17observed
    crypto-common0.1.7observed
    csv direct1.4.0observed
    csv-core0.1.13observed
    deranged0.5.8observed
    digest0.10.7observed
    dirs5.0.1observed
    dirs-sys0.4.1observed
    either1.19.0observed
    encode_unicode1.0.0observed
    equivalent1.0.2observed
    errno0.3.14observed
    fastrand2.5.0observed
    filetime0.2.29observed
    find-msvc-tools0.1.14observed
    flate21.1.10observed
    fnv1.0.7observed
    foreign-types0.3.2observed
    foreign-types-shared0.1.1observed
    form_urlencoded1.2.2observed
    fs20.4.3observed
    futures-channel0.3.34observed
    futures-core0.3.34observed
    futures-io0.3.34observed
    futures-sink0.3.34observed
    futures-task0.3.34observed
    futures-util0.3.34observed
    getrandom0.2.17observed
    glob0.3.4observed
    h20.3.27observed
    hashbrown direct0.14.5observed
    hashbrown0.17.1observed
    hmac0.12.1observed
    http0.2.12observed
    http-body0.4.6observed
    httpdate1.0.3observed
    hyper0.14.32observed
    hyper-tls0.5.0observed
    icu_collections2.3.0observed
    icu_locale_core2.3.0observed
    icu_normalizer2.3.0observed
    icu_properties2.3.0observed
    icu_provider2.3.1observed
    idna1.1.0observed
    idna_adapter1.2.2observed
    indexmap2.14.2observed
    indicatif0.16.2observed
    inout0.1.4observed
    ipnet2.12.2observed
    itertools direct0.11.0observed
    itoa1.0.18observed
    jobserver0.1.35observed
    js-sys0.3.106observed
    lazy_static direct1.5.1observed
    libredox0.1.25observed
    linux-raw-sys0.12.1observed
    litemap0.8.3observed
    log0.4.34observed
    memchr2.8.3observed
    mime0.3.17observed
    miniz_oxide0.9.1observed
    mio1.2.4observed
    num-conv0.2.2observed
    number_prefix0.4.0observed
    once_cell1.21.4observed
    openssl-probe0.2.1observed
    option-ext0.2.0observed
    password-hash0.4.2observed
    pbkdf20.11.0observed
    percent-encoding2.3.2observed
    pin-project-lite0.2.17observed
    pkg-config0.3.34observed
    potential_utf0.1.6observed
    powerfmt0.2.1observed
    ppv-lite860.2.21observed
    r-efi6.0.0observed
    rand0.8.8observed
    rand_chacha0.3.1observed
    rand_core0.6.4observed
    rayon direct1.12.0observed
    redox_users0.4.6observed
    regex direct1.13.1observed
    regex-automata0.4.18observed
    regex-syntax0.8.11observed
    reqwest0.11.27observed
    rustls-pemfile1.0.4observed
    ryu1.0.23observed
    schannel0.1.29observed
    scopeguard1.2.0observed
    security-framework3.7.0observed
    security-framework-sys2.17.0observed
    serde_urlencoded0.7.1observed
    sha10.10.7observed
    sha20.10.9observed
    shlex2.0.1observed
    simd-adler320.3.10observed
    simdutf80.1.5observed
    slab0.4.12observed
    smallvec1.16.2observed
    socket20.5.10observed
    socket20.6.5observed
    stable_deref_trait1.2.1observed
    subtle2.6.1observed
    syn2.0.119observed
    syn3.0.6observed
    sync_wrapper0.1.2observed
    synstructure0.14.0observed
    system-configuration0.5.1observed
    tar0.4.46observed
    tempfile3.27.0observed
    time0.3.55observed
    time-core0.1.9observed
    tinystr0.8.4observed
    tinyvec1.13.3observed
    tokio1.53.2observed
    tokio-native-tls0.3.1observed
    tokio-util0.7.19observed
    tower-service0.3.3observed
    tracing0.1.44observed
    tracing-core0.1.36observed
    try-lock0.2.5observed
    typenum1.20.1observed
    unicode-ident1.0.26observed
    unicode-normalization direct0.1.25observed
    unicode-normalization-alignments direct0.1.12observed
    url2.5.8observed
    utf8_iter1.0.4observed
    version_check0.9.5observed
    want0.3.2observed
    wasi0.11.1+wasi-snapshot-preview1observed
    wasm-bindgen-futures0.4.79observed
    wasm-bindgen-macro-support0.2.129observed
    web-sys0.3.106observed
    windows-link0.2.1observed
    windows-sys0.48.0observed
    windows-sys0.52.0observed
    windows-sys0.61.2observed
    windows-targets0.48.5observed
    windows-targets0.52.6observed
    winreg0.50.0observed
    writeable0.6.4observed
    xattr1.6.1observed
    yoke0.8.3observed
    zerofrom0.1.8observed
    zerotrie0.2.5observed
    zerovec0.11.8observed
    zip0.6.6observed
    zlib-rs0.6.8observed
    zstd0.11.2+zstd.1.5.2observed

    1 of 1 jobs captured: build-test success · Generated 2026-10-08 · tarp 0.1.0+20261008141706-g2aea6e37e0b6 · run bc10c9e9-0736-4d1e-94fa-79d1c90917b7