Robots.txt Monitoring

Validates your robots.txt file and monitors it for changes. An incorrect robots.txt can accidentally block search engines from indexing your site or expose paths you intended to keep private.

How It Works

Hoppla fetches and parses your robots.txt file, checking its syntax against the Robots Exclusion Standard. It validates directives, detects potential issues, and alerts you when the file changes unexpectedly.

What Hoppla Checks

• File availability -- robots.txt is accessible and returns HTTP 200
• User-agent groups -- Parses each user-agent section and its directives
• AI bot access -- Classifies known AI crawlers as blocked, allowed, or not mentioned (see below)
• SEO issues -- Detects typed issues such as a missing sitemap, blanket Disallow: /, a fully blocked Googlebot, and blocked CSS/JS paths
• Sitemap references -- Extracts Sitemap: directives
• Change detection -- Computes a content hash and surfaces a diff when the file changes between checks

AI Bot Access

The robots.txt check classifies known AI crawlers by how your file treats them — blocked, allowed, or not mentioned. The tracked bot families include:

• OpenAI — GPTBot, OAI-SearchBot, ChatGPT-User
• Anthropic — ClaudeBot, Claude-Web, Claude-SearchBot, Claude-User, Anthropic
• Perplexity — PerplexityBot, Perplexity-User
• Google — Google-Extended
• Common Crawl — CCBot
• ByteDance — Bytespider
• Meta — FacebookBot, meta-externalagent
• Amazon — Amazonbot
• Apple — Applebot-Extended

When a check newly blocks or unblocks one of these bots, the change is surfaced in the diff (ai_bot_blocked / ai_bot_unblocked transitions). This is the centrepiece of the check for GEO / AI readiness.

Configuration

Setting	Description	Default
Check interval	How often to validate the file	12 hours

The robots.txt URL is fixed at {monitor_url}/robots.txt and is not a configurable setting.

Results

Field	Description
File present (has_robots_txt)	Whether robots.txt was found
Content hash	SHA-256 of the file, used to detect changes
User-agent groups	Each user-agent section and its directives
AI bot classification	Bots split into blocked, allowed, and not_mentioned
Sitemaps	URLs referenced in Sitemap: directives
SEO issues	Typed issues: missing_sitemap, blanket_disallow (critical), googlebot_blocked (critical), css_js_blocked
Changes	Diff against the previous check (including AI bot block/unblock transitions)

Example Result

Robots.txt: https://example.com/robots.txt (200 OK)

  User-agent groups:  3
  Sitemaps:
    - https://example.com/sitemap.xml
    - https://example.com/sitemap-news.xml

  AI bots:
    Blocked:        GPTBot, CCBot
    Allowed:        ClaudeBot
    Not mentioned:  PerplexityBot, Google-Extended

  Issues:
    - googlebot_blocked (critical): Googlebot is completely blocked with Disallow: /
    - missing_sitemap (warning): No Sitemap directive found in robots.txt

Alerts

Event	Trigger	Severity
warning.robots_txt	File is missing, has syntax errors, contains risky directives, or changes unexpectedly	Warning

The alert includes details about what changed or what issues were found. Change detection alerts show a diff of the previous and current file content.

Common Issues

• Disallow: / for all user-agents -- Blocks all search engine crawling entirely
• Missing robots.txt -- Returns 404, which means no crawl restrictions (may be intentional)
• Sitemap URL returning errors -- The sitemap referenced in robots.txt is not accessible
• Accidental changes -- A deployment or CMS update overwrites robots.txt with incorrect rules
• Typos in directives -- Misspelled directives like Dissallow are silently ignored by crawlers