← All skills

webcrawler-deep-crawl

95 / 100 · Low risk

Fine to try.

By browser-act/skills (GitHub) · coding · 6,341 stars · scanned at commit 593316a

Deep-crawl any website from start URLs, return per-page LLM-ready text/markdown/HTML plus metadata (title, description, author, language, canonical URL, OG) and in-scope outbound links. Use when user mentions deep crawl website, recursive crawl, crawl a whole site, scrape entire website, scrape docs site, scrape documentation, scrape knowledge base, scrape blog, build RAG corpus, build vector database from website, knowledge base for chatbot, GPT knowledge files, llms.txt, sitemap crawl, BFS cr…

Works with any agent that supports SKILL.md skills.

What this skill can do

Install

Paste this into your terminal. It downloads exactly the version we scanned, not whatever is newest.

if [ -e ~/.agents/skills/'webcrawler-deep-crawl' ] || [ -L ~/.agents/skills/'webcrawler-deep-crawl' ]; then echo 'A skill folder named webcrawler-deep-crawl already exists — not replacing it' >&2; false; else d=$(mktemp -d) && git clone --quiet --filter=blob:none --no-checkout 'https://github.com/browser-act/skills' "$d" && git -C "$d" checkout --quiet '593316aa0f8ccd88bdf745c92b24a50641701973' -- 'solutions/search-research/webcrawler-deep-crawl' && mkdir -p ~/.agents/skills && cp -R "$d"/'solutions/search-research/webcrawler-deep-crawl' ~/.agents/skills/'webcrawler-deep-crawl' && rm -rf "$d"; fi

Codex reads ~/.agents/skills directly. For Claude Code, Cursor, Copilot, Gemini CLI, Antigravity or OpenCode, also link it. For Claude Code: ln -s ~/.agents/skills/'webcrawler-deep-crawl' ~/.claude/skills/'webcrawler-deep-crawl' (other agents: ~/.cursor/skills, ~/.copilot/skills, ~/.gemini/skills, ~/.gemini/config/skills, ~/.config/opencode/skills). Or install through the finder skill, which does both.

Badge for your README

The badge for browser-act/skills shows the score of its lowest-scoring skill.

[![SkillTrust](https://zalavijaysinh.github.io/skilltrust/badges/browser-act/skills.svg)](https://zalavijaysinh.github.io/skilltrust/repos/browser-act/skills/)

Score history