← All skills

crawling-a-site

95 / 100 · Low risk

Fine to try. Some of what it can do is expected: a coding skill usually runs build and test commands.

By hashgraph-online/awesome-codex-plugins (GitHub) · coding · 1,568 stars · scanned at commit 3e1e6b2

Use when the user wants to follow links across a domain and capture every reachable page as Markdown. Covers `crawlberg crawl` with depth, page caps, concurrency, rate limiting, domain scoping, robots, and output selection.

Works with any agent that supports SKILL.md skills.

What this skill can do

Other copies in this repository

hashgraph-online/awesome-codex-plugins has 1 more folder with a skill named crawling-a-site. This page shows the lowest-scoring copy.

• plugins/kreuzberg-dev/plugins/plugins/kreuzcrawl/skills/crawling-a-site 95/100 · Low risk

Install

Paste this into your terminal. It downloads exactly the version we scanned, not whatever is newest.

if [ -e ~/.agents/skills/'crawling-a-site' ] || [ -L ~/.agents/skills/'crawling-a-site' ]; then echo 'A skill folder named crawling-a-site already exists — not replacing it' >&2; false; else d=$(mktemp -d) && git clone --quiet --filter=blob:none --no-checkout 'https://github.com/hashgraph-online/awesome-codex-plugins' "$d" && git -C "$d" checkout --quiet '3e1e6b20ffc28f42923e391f29f075c424e19e64' -- 'plugins/kreuzberg-dev/plugins/plugins/crawlberg/skills/crawling-a-site' && mkdir -p ~/.agents/skills && cp -R "$d"/'plugins/kreuzberg-dev/plugins/plugins/crawlberg/skills/crawling-a-site' ~/.agents/skills/'crawling-a-site' && rm -rf "$d"; fi

Codex reads ~/.agents/skills directly. For Claude Code, Cursor, Copilot, Gemini CLI, Antigravity or OpenCode, also link it. For Claude Code: ln -s ~/.agents/skills/'crawling-a-site' ~/.claude/skills/'crawling-a-site' (other agents: ~/.cursor/skills, ~/.copilot/skills, ~/.gemini/skills, ~/.gemini/config/skills, ~/.config/opencode/skills). Or install through the finder skill, which does both.

Badge for your README

The badge for hashgraph-online/awesome-codex-plugins shows the score of its lowest-scoring skill.

[![SkillTrust](https://zalavijaysinh.github.io/skilltrust/badges/hashgraph-online/awesome-codex-plugins.svg)](https://zalavijaysinh.github.io/skilltrust/repos/hashgraph-online/awesome-codex-plugins/)

Score history