← All skills

algo-seo-crawl

95 / 100 · Low risk

Fine to try.

By charlieviettq/awesome-agent-skill (GitHub) · coding · 27 stars · scanned at commit c567ae4

Implement a web crawler pipeline covering URL discovery, fetching, parsing, and storage. Use this skill when the user needs to build a site crawler, audit website structure, or collect web data systematically — even if they say 'scrape a website', 'crawl all pages', or 'site audit spider'.

Works with any agent that supports SKILL.md skills.

What this skill can do

Install

Paste this into your terminal. It downloads exactly the version we scanned, not whatever is newest.

if [ -e ~/.agents/skills/'algo-seo-crawl' ] || [ -L ~/.agents/skills/'algo-seo-crawl' ]; then echo 'A skill folder named algo-seo-crawl already exists — not replacing it' >&2; false; else d=$(mktemp -d) && git clone --quiet --filter=blob:none --no-checkout 'https://github.com/charlieviettq/awesome-agent-skill' "$d" && git -C "$d" checkout --quiet 'c567ae426d55fa498e919fc0f5a37c582d5c7a2c' -- '.cursor/skills/asgard-ai-platform/algo-seo-crawl' && mkdir -p ~/.agents/skills && cp -R "$d"/'.cursor/skills/asgard-ai-platform/algo-seo-crawl' ~/.agents/skills/'algo-seo-crawl' && rm -rf "$d"; fi

Codex reads ~/.agents/skills directly. For Claude Code, Cursor, Copilot, Gemini CLI, Antigravity or OpenCode, also link it. For Claude Code: ln -s ~/.agents/skills/'algo-seo-crawl' ~/.claude/skills/'algo-seo-crawl' (other agents: ~/.cursor/skills, ~/.copilot/skills, ~/.gemini/skills, ~/.gemini/config/skills, ~/.config/opencode/skills). Or install through the finder skill, which does both.

Badge for your README

The badge for charlieviettq/awesome-agent-skill shows the score of its lowest-scoring skill.

[![SkillTrust](https://zalavijaysinh.github.io/skilltrust/badges/charlieviettq/awesome-agent-skill.svg)](https://zalavijaysinh.github.io/skilltrust/repos/charlieviettq/awesome-agent-skill/)

Score history