lance-tokenizer cargo 13.0.0 clean Tokenizer abstractions and implementations for Lance 2M 21h ago dynamo-tokenizers cargo 3.1.0 clean Standalone HuggingFace, tiktoken, fastokens, and Baseten tokenizer implementations for LLM inference serving. 748K 1d ago turbojev-tokenizer cargo 0.31.0 clean Tokenizer-safe candidate code discovery for TurboJev backends 258 4h ago lt-tokenize cargo 0.2.8 clean Sentence splitting and word tokenization with UTF-8 byte offsets 228 2d ago summa-tokenizer cargo 2.1.0 clean Stable-Rust byte-level BPE tokenization for Summa 85 16h ago gproxy-tokenizer cargo 4.1.1 clean Runtime-independent local token counting for GPROXY 53 23h ago bpe cargo 0.2.3 clean Fast byte-pair encoding implementation. 251K 3h ago bpe-openai cargo 0.3.2 clean Prebuilt fast byte-pair encoders for OpenAI. 237K 4h ago css_lexer cargo 0.0.35-canary.830daadacc clean A spec-compliant CSS tokenizer with zero-copy cursors and optional feature gates. 92K 3h ago apr-format cargo 0.70.2 clean Sovereign `.apr` model container format (read + write, v1 APRN + v2 APR\0) — zero ML/GPU/tokenizer dependencies 28K 11h ago formualizer-parse cargo 3.2.0 clean High-performance Excel/OpenFormula tokenizer + parser with a stable AST surface 14K 18h ago cortiq-core cargo 0.8.13 clean CMF (Cortiq Model Format): a self-describing, memory-mappable binary container for quantized LLM weights, tokenizer, per-task masks and per-skill delta records. 4.8K 1d ago cortiq-engine cargo 0.8.13 clean Portable inference runtime for the CMF model format, with no ML framework underneath: runs on CPU, and on GPU (Vulkan / Metal / DX12) with the `gpu` feature;… 4.4K 1d ago edifact-parser cargo 0.30.0 clean Streaming EDIFACT tokenizer and SAX-style parser — standalone, no BO4E dependency 4.2K 8h ago socketry-markdown cargo 0.6.0 clean CommonMark compliant markdown parser in Rust with ASTs and extensions 2.1K 2d ago uqa-analysis cargo 0.5.1 clean Tokenizers, char/token filters, and analyzers for UQA full-text search 1.3K 10h ago frost-lexer cargo 0.1.24 clean Zsh-compatible lexer/tokenizer 653 20h ago suiko-sudachi cargo 0.7.0 clean Unofficial crates.io redistribution of WorksApplications/sudachi.rs v0.6.11 (Japanese morphological analyzer) so that the suiko CLI can be installed from… 637 23h ago monostyle_lexer cargo 0.3.26 clean Profile-driven tokenizer that extracts layout and complexity signals from source code. 620 21h ago bunsen-bundled-whisper cargo 0.38.0 clean Bunsen bundled Whisper assets: fetches, pins and caches OpenAI's checkpoint, its tokenizer vocabularies, and the ONNX-generated reference model. Hosts none. 233 22h ago turbojev-runtime-conformance cargo 0.31.0 clean Portable conformance kit for TurboJev tokenizer and runtime adapters 182 3h ago sil-lexer cargo 0.7.0 clean Silc lexical analysis (Raku-inspired surface) 18 2d ago tokwc cargo 0.1.0 clean local wc-style token counter for OpenAI and Claude models 8 1d ago 