Broken Link Detection

Broken link detection crawls your website to find links that return HTTP errors -- 404 Not Found, 5xx server errors, timeouts, and other failures. Broken links frustrate visitors, hurt SEO rankings, and signal neglected content.

How It Works

When a broken links check is triggered, Hoppla crawls your site. The crawler starts at your monitor's URL and follows internal links, checking each page it discovers for broken outbound links. Only one crawl runs at a time per monitor -- if a crawl is already in progress, the check is skipped.

Crawl Process

1. The crawler starts at your monitor's URL (or a sitemap, see below) and discovers pages by following same-domain links.
2. For each page, all links (internal and external) are checked for HTTP errors.
3. Results are aggregated: total pages crawled, broken links found, and response times.

Same-Domain Restriction

The crawler only follows links to pages on the same domain as your monitor URL. External links are checked for reachability (status code) but are not crawled further. This prevents the crawler from spiraling out to the entire internet.

Files and Downloads

Links to files (PDFs, images, videos, archives) are checked by their HTTP status alone. The crawler reads the response headers and ends the transfer before the file itself is downloaded, so a large file costs about as much as a page and cannot time out halfway through its download. Pages, text and XML are downloaded as before, so their links can be followed.

A Second Try Before a Link Counts as Broken

A link that could not be fetched during the crawl (a timeout, a dropped connection, or a 408, 429 or 5xx answer) is requested once more on its own after the crawl, with up to 30 seconds to answer. If it answers then, it is not reported. If it fails again, it is reported with the newer answer. A definite answer such as 404, 410 or 403 is not asked again. Up to 20 links per crawl get a second try.

Cloudflare WAF Handling

Sites behind Cloudflare's Web Application Firewall (WAF) may block the crawler's bot User-Agent. Hoppla includes automatic retry middleware that:

1. Detects Cloudflare blocks (HTTP 403/429 with Cloudflare headers, or 520-530 status codes).
2. Retries the request with browser-like headers (User-Agent, Accept, Sec-Fetch headers).
3. If the retry succeeds, switches all subsequent requests to browser headers for the remainder of the crawl.
4. If the retry still fails, marks the URL as WAF-blocked rather than broken.

WAF-blocked URLs are tracked separately in the results so you can distinguish between genuinely broken links and Cloudflare-protected pages.

Configuration

Setting	Default	Range	Description
Max pages	500	10 - 5,000	Maximum number of pages to crawl per run.
Max depth	10	1 - 10	Maximum link depth from the starting URL.
Concurrency	2	1 - 5	Number of simultaneous HTTP requests during the crawl.
Requests per second	2	0.5 - 10	Rate limit to avoid overwhelming your server.
Page timeout	10s	1 - 30s	Maximum time to wait for each page response.
Respect robots.txt	Yes	--	Whether to honor your site's robots.txt directives.
Detect mixed content	Yes	--	Whether to also check for HTTP resources on HTTPS pages during the crawl.
Seed from sitemap	Off	--	Start the crawl from the URLs listed in your sitemap instead of (or in addition to) following links from the start page.
Include / exclude patterns	(none)	--	Force specific URLs to always be crawled, or ignore URL patterns you want the crawler to skip (e.g. faceted-search or calendar pages).
Crawler User-Agent	HopplaBot/1.0	--	The User-Agent the crawler identifies itself with. You can set a custom value -- useful for allow-listing the crawler through a firewall.

Result Retention

Hoppla keeps the most recent crawl results per monitor and prunes older ones automatically.

Results

Each broken links check result includes:

• Status -- ok if no broken links found, warning if broken links were detected
• Pages crawled -- Total number of pages the crawler visited
• Broken links total -- Count of broken links found across all pages
• Mixed content total -- Count of mixed content issues found (if enabled)
• WAF blocked total -- Count of URLs blocked by Cloudflare WAF
• Average response time -- Average page response time across all crawled pages

Broken Link Details

For each broken link (up to 100), the result includes:

• Source page -- The page that contains the broken link
• Target URL -- The URL that is broken
• HTTP status code -- The error code returned (404, 500, etc.). A link that gave no HTTP answer shows what went wrong instead: Timeout, DNS error, Connection error or TLS error.

CSV Export

The Download CSV button in the report exports every broken link of the latest crawl, not only the first 100 shown, with its status, the reason, the page it was found on and its link text. Use it to hand the list to whoever maintains the site. For German-language accounts the file is separated by semicolons, as German Excel expects; otherwise by commas.

Alerts

Broken links is an advisory check -- it produces warnings but never creates incidents.

Warning Notification

A warning.broken_links notification is sent when the crawl finds one or more broken links. The notification includes the count of broken links and the number of pages crawled.

Notification Events

Event	When
warning.broken_links	One or more broken links found during the crawl
warning.cleared	A subsequent crawl found no broken links

Tips

• Start with a low max pages value (e.g., 100) to get a quick overview, then increase it for thorough scans.
• Lower the concurrency and requests per second if the crawl is causing load on your server.
• Lower max pages and max depth to avoid spidering large dynamic or paginated sections that generate many similar pages.
• Check the WAF blocked count -- if it is high, your Cloudflare rules may need a user-agent exception. The default crawler User-Agent is HopplaBot/1.0 (+https://hoppla.dev/bot). White-label and reseller deployments crawl under a custom User-Agent, so allow-list rules should use the exact value shown in the monitor's crawler settings.