← All skills

llm-crawler-access-check

100 / 100 · Low risk

Fine to try.

By davepoon/buildwithclaude (GitHub) · coding · 3,743 stars · scanned at commit 941b335

Check whether a website's robots.txt allows the AI crawlers that decide visibility in ChatGPT Search, Perplexity, Claude, Gemini, and Microsoft Copilot. Use when someone asks whether AI bots are blocked, whether to allow or block GPTBot, why a site never appears in AI answers, or wants a robots.txt review for AI crawlers. Reads only robots.txt, then returns a per-agent allow/block table, the exact rule responsible for each verdict, and the precise lines to change. Distinguishes training crawler…

Works with any agent that supports SKILL.md skills.

What this skill can do

We found nothing in the files we read that matches our rules. That does not prove the skill is safe: see the limits of static scanning.

Install

Paste this into your terminal. It downloads exactly the version we scanned, not whatever is newest.

if [ -e ~/.agents/skills/'llm-crawler-access-check' ] || [ -L ~/.agents/skills/'llm-crawler-access-check' ]; then echo 'A skill folder named llm-crawler-access-check already exists — not replacing it' >&2; false; else d=$(mktemp -d) && git clone --quiet --filter=blob:none --no-checkout 'https://github.com/davepoon/buildwithclaude' "$d" && git -C "$d" checkout --quiet '941b335e8e2d748d9855f77c0fc0aaec924b75b9' -- 'plugins/all-skills/skills/llm-crawler-access-check' && mkdir -p ~/.agents/skills && cp -R "$d"/'plugins/all-skills/skills/llm-crawler-access-check' ~/.agents/skills/'llm-crawler-access-check' && rm -rf "$d"; fi

Codex reads ~/.agents/skills directly. For Claude Code, Cursor, Copilot, Gemini CLI, Antigravity or OpenCode, also link it. For Claude Code: ln -s ~/.agents/skills/'llm-crawler-access-check' ~/.claude/skills/'llm-crawler-access-check' (other agents: ~/.cursor/skills, ~/.copilot/skills, ~/.gemini/skills, ~/.gemini/config/skills, ~/.config/opencode/skills). Or install through the finder skill, which does both.

Badge for your README

The badge for davepoon/buildwithclaude shows the score of its lowest-scoring skill.

[![SkillTrust](https://zalavijaysinh.github.io/skilltrust/badges/davepoon/buildwithclaude.svg)](https://zalavijaysinh.github.io/skilltrust/repos/davepoon/buildwithclaude/)

Score history