How alerts work

Everguardly's alert pipeline has three stages: detection, fan-out, and delivery. Understanding how they fit together is the difference between "alerts are noisy" and "alerts are useful".

1. Detection

Detection happens in the worker process. A scheduler ticks every 30 seconds and queues HTTP checks for any monitor whose lastCheckAt + intervalSeconds has lapsed. SSL and domain checks follow a separate hourly tick gated to once per 24 hours per monitor.

When a check fails, we don't immediately raise an alert. We retry once after a few seconds. If the retry also fails, the worker opens an incident row and the alert pipeline kicks in. That single-retry confirmation is what prevents flaky-network false alarms.

2. Fan-out

Fan-out is the worker's alert-worker process. It receives a queue job with monitorId + incidentId + event type (down / up / cert_expiring / domain_expiring), looks up the routing — which channels should receive this alert for this monitor — and queues one delivery job per channel.

The routing chain is:

1. Monitor-level channels — if any are set in the Edit monitor dialog
2. Otherwise, all default channels — every row in Settings → Channels with isDefault=true

Dedup happens here too. Within 5 minutes, we won't fire the same alert type for the same incident on the same channel twice. Reminder alerts (cert expiring in N days) get a stronger dedup via a unique-key index on (kind, monitorId, trigger, expiresAt) so re-running the scheduler is always safe.

3. Delivery

Delivery is one process per channel type. Email is delivered through our transactional email provider. Slack and Discord both use webhook POST. Generic webhooks get a JSON envelope with the event type, monitor, and payload.

Each delivery is logged in the alerts table with status: pending → sent or failed. The Test button on the channels page hits the same delivery code path with a synthetic payload, so a passing test really does prove the production path works.

That's the whole pipeline

Three stages, three queues, one dedup key per stage.