← All skills

huggingface-tokenizers

80 / 100 · Low risk

Fine to try. Some of what it can do is expected: a coding skill usually runs build and test commands.

By hashgraph-online/awesome-codex-plugins (GitHub) · coding · 1,573 stars · scanned at commit 70b6a3a

Provides the HuggingFace Tokenizers library (Rust core with Python and Node.js bindings) for training and using BPE, WordPiece, and Unigram tokenizers. Covers the pipeline of normalizers, pre-tokenizers, models, post-processors and decoders, padding and truncation, batch encoding, alignment tracking, and conversion to transformers PreTrainedTokenizerFast. Use when training a custom tokenizer or vocabulary on a new corpus, tokenizing large text corpora quickly, mapping tokens back to character o…

Works with any agent that supports SKILL.md skills.

What this skill can do

Install

Paste this into your terminal. It downloads exactly the version we scanned, not whatever is newest.

if [ -e ~/.agents/skills/'huggingface-tokenizers' ] || [ -L ~/.agents/skills/'huggingface-tokenizers' ]; then echo 'A skill folder named huggingface-tokenizers already exists — not replacing it' >&2; false; else d=$(mktemp -d) && git clone --quiet --filter=blob:none --no-checkout 'https://github.com/hashgraph-online/awesome-codex-plugins' "$d" && git -C "$d" checkout --quiet '70b6a3a3a7c774671e45af646306e1448baf0511' -- 'plugins/KalarisLabs/research-agent-skills/skills/huggingface-tokenizers' && mkdir -p ~/.agents/skills && cp -R "$d"/'plugins/KalarisLabs/research-agent-skills/skills/huggingface-tokenizers' ~/.agents/skills/'huggingface-tokenizers' && rm -rf "$d"; fi

Codex reads ~/.agents/skills directly. For Claude Code, Cursor, Copilot, Gemini CLI, Antigravity or OpenCode, also link it. For Claude Code: ln -s ~/.agents/skills/'huggingface-tokenizers' ~/.claude/skills/'huggingface-tokenizers' (other agents: ~/.cursor/skills, ~/.copilot/skills, ~/.gemini/skills, ~/.gemini/config/skills, ~/.config/opencode/skills). Or install through the finder skill, which does both.

Badge for your README

The badge for hashgraph-online/awesome-codex-plugins shows the score of its lowest-scoring skill.

[![SkillTrust](https://zalavijaysinh.github.io/skilltrust/badges/hashgraph-online/awesome-codex-plugins.svg)](https://zalavijaysinh.github.io/skilltrust/repos/hashgraph-online/awesome-codex-plugins/)

Score history