← methodology

robots.txt present & parseable

access.robots-txt · Access · 2 pts · required

Goal

Do you welcome agents?

Whether your robots policy, bot protection, and agent guidance actively admit AI agents instead of blocking them.

How we test

Fetched /robots.txt and validated it parses as a robots policy.

How to pass

Serve a valid /robots.txt; without one, agent crawlers guess your policy.

Spec

https://www.rfc-editor.org/rfc/rfc9309 ↗

Verify a fix

curl -X POST /api/scan/checks \
  -H 'content-type: application/json' \
  -d '{"url": "yourdomain.com", "checkIds": ["access.robots-txt"]}'