← All skills

tokenization-sentencepiece

80 / 100 · Low risk

Fine to try. Some of what it can do is expected: a coding skill usually runs build and test commands.

By davila7/claude-code-templates (GitHub) · coding · 33,256 stars · scanned at commit a6e6f93

Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.

Works with any agent that supports SKILL.md skills.

What this skill can do

Install

Paste this into your terminal. It downloads exactly the version we scanned, not whatever is newest.

if [ -e ~/.agents/skills/'tokenization-sentencepiece' ] || [ -L ~/.agents/skills/'tokenization-sentencepiece' ]; then echo 'A skill folder named tokenization-sentencepiece already exists — not replacing it' >&2; false; else d=$(mktemp -d) && git clone --quiet --filter=blob:none --no-checkout 'https://github.com/davila7/claude-code-templates' "$d" && git -C "$d" checkout --quiet 'a6e6f93e9a4baaa6ff337aee5505bf5c2bae4ce6' -- 'cli-tool/components/skills/ai-research/tokenization-sentencepiece' && mkdir -p ~/.agents/skills && cp -R "$d"/'cli-tool/components/skills/ai-research/tokenization-sentencepiece' ~/.agents/skills/'tokenization-sentencepiece' && rm -rf "$d"; fi

Codex reads ~/.agents/skills directly. For Claude Code, Cursor, Copilot, Gemini CLI, Antigravity or OpenCode, also link it. For Claude Code: ln -s ~/.agents/skills/'tokenization-sentencepiece' ~/.claude/skills/'tokenization-sentencepiece' (other agents: ~/.cursor/skills, ~/.copilot/skills, ~/.gemini/skills, ~/.gemini/config/skills, ~/.config/opencode/skills). Or install through the finder skill, which does both.

Badge for your README

The badge for davila7/claude-code-templates shows the score of its lowest-scoring skill.

[![SkillTrust](https://zalavijaysinh.github.io/skilltrust/badges/davila7/claude-code-templates.svg)](https://zalavijaysinh.github.io/skilltrust/repos/davila7/claude-code-templates/)

Score history