Incidents

An incident represents an ongoing problem with a monitored service. Incidents are created automatically when availability checks fail and resolved automatically when they pass again.

Incident Lifecycle

Every incident moves through a defined lifecycle:

1. Created -- A check detected a failure. Notifications are sent to matching channels.
2. Acknowledged (optional) -- A team member acknowledged the incident, indicating someone is looking into it. This stops repeated notifications.
3. Resolved -- The check passed again (or the incident was manually resolved). A resolution notification is sent.

Each state change is recorded with a timestamp and displayed on the incident timeline.

Automatic Creation

Incidents are created automatically when the following check types detect failures:

• Uptime -- HTTP response errors or timeouts (verified from a second probe location first).
• SSL -- Expired or invalid certificates.
• DNS -- Failed DNS resolution.
• Cron monitors -- Missed heartbeat pings.
• Agent monitors -- Missed agent heartbeats.
• Application health -- Health endpoint returning unhealthy status.

Only one open incident exists per monitor at a time. If a monitor already has an open incident and the check fails again, the existing incident is updated rather than creating a duplicate.

Automatic Resolution

When a previously failing check passes again, the open incident is automatically resolved. The resolution includes:

• The time the check recovered.
• Total incident duration.
• A resolution notification sent through the same channels that received the creation alert.

Detection sensitivity and tuning

Hoppla deliberately errs toward not paging you for momentary blips. Several layers sit between a single failed check and an alert reaching you — and each is adjustable per monitor.

Dual-check verification (uptime)

A failed uptime check is immediately re-run from a second probe location before it counts. A failure seen from only one location is treated as a glitch near that probe, not an outage, and never opens an incident. (Self-hosted single-probe installs are warned by hoppla:doctor, because the second opinion then comes from the same place.)

Confirmation threshold — how quickly an incident opens

An incident opens only after confirmation_threshold consecutive confirmed failures (default 2). A single failed check never opens an incident.

Notification grace — sub-minute blips stay silent

When an incident opens, the outbound notification is held for alert_grace_seconds (default 60). The incident is recorded immediately (visible on the dashboard and status page), but if the check recovers within the grace window, neither a down nor a recovery email is sent — a 30-second blip never reaches your inbox. Set this to 0 to page the instant an incident opens.

Recovery threshold — how an incident resolves

An open incident resolves only after recovery_threshold consecutive passing checks (default 2). This hysteresis stops a single lucky success in the middle of an unstable period from prematurely closing an incident that then immediately re-opens.

Flapping — one notification instead of a storm

If a monitor oscillates up and down repeatedly (3 or more changes between up and down within 10 minutes), Hoppla recognises it as flapping: it suppresses the individual incident/recovery alerts and sends a single “monitor is flapping” notification instead. Warnings don't count: a slow response or a redirect finding that comes and goes stays a warning. Once the monitor holds steady for 5 minutes, flapping clears and the next check sends a “flapping resolved” notice. This is what keeps an unstable target from filling your inbox with paired down/up emails.

Cooldown — rate-limit repeat alerts

cooldown_seconds (default 0, off) sets a minimum gap between notifications for the same monitor, independent of state changes.

Adjusting the thresholds

All of the above are per-monitor and live under Monitor → Notifications in the dashboard (or the notification_settings object on the monitor API):

Setting	Default	Range	Effect
Confirmation threshold	2	1–10	consecutive failures before an incident opens
Recovery threshold	2	1–10	consecutive passes before an incident resolves
Alert grace (seconds)	60	0–3600	holds the down-notification; sub-grace blips never page
Cooldown (seconds)	0	0–86400	minimum gap between repeat notifications

Tuning examples

• Page on the first failure (a critical service where every second counts): Confirmation threshold 1, Alert grace 0.
• Only sustained outages should page (a noisy or edge service that blips often): raise Alert grace to 120 and Recovery threshold to 3 so brief instability is ridden out silently.

Defaults are chosen so a genuine multi-minute outage still pages promptly while a sub-minute blip or a flapping target does not. Lower the thresholds for latency-sensitive services; raise them for noisy ones.

Manual Actions

From the incident detail page, team members can:

• Acknowledge -- Mark the incident as seen. This stops repeated escalation notifications while you investigate. The incident remains open until the check passes or you resolve it manually.
• Snooze / Un-snooze -- Temporarily suppress repeat notifications for this incident until a chosen time. Like acknowledging, but time-boxed -- snoozed incidents skip repeat notifications, though the resolution notification still fires when the check recovers. You can un-snooze an incident at any time to resume notifications immediately.
• Archive / Unarchive -- Move a resolved or old incident out of the active list to keep your incident view focused. Archived incidents can be unarchived at any time.
• Add notes -- Attach free-text notes to the incident timeline. Use this to document what you found, what you did, or any context for the team.

Incident Detail Page

The incident detail page shows everything about a specific incident:

• Cause -- What triggered the incident (e.g., "HTTP 503 Service Unavailable", "Certificate expired", "Ping not received").
• Duration -- How long the incident lasted (or is ongoing).
• Timeline -- Chronological log of all state changes (created, acknowledged, notes added, resolved) with timestamps.
• Affected Monitor -- Link to the monitor that generated the incident.
• Notification Log -- Which notifications were sent, to which channels, and when.