← methodology
AI crawler policy
access.ai-crawlers · Access · 5 pts · required
Goal
Do you welcome agents?
Whether your robots policy, bot protection, and agent guidance actively admit AI agents instead of blocking them.
How we test
Evaluated robots.txt groups for the major AI agent user-agents (GPTBot, ClaudeBot, PerplexityBot, …).
How to pass
Add explicit User-agent groups allowing the AI agents you welcome; a blanket Disallow makes you invisible to them.
Verify a fix
curl -X POST /api/scan/checks \
-H 'content-type: application/json' \
-d '{"url": "yourdomain.com", "checkIds": ["access.ai-crawlers"]}'