How alerts work

Everguardly's alert pipeline has three stages: detection, fan-out, and delivery. Understanding how they fit together is the difference between "alerts are noisy" and "alerts are useful".

1. Detection

Detection happens in the worker process. A scheduler ticks every 30 seconds and queues HTTP checks for any monitor whose lastCheckAt + intervalSeconds has lapsed. SSL and domain checks follow a separate hourly tick gated to once per 24 hours per monitor.

When a check fails, we don't immediately raise an alert. We retry once after a few seconds. If the retry also fails, the worker opens an incident row and the alert pipeline kicks in. That single-retry confirmation is what prevents flaky-network false alarms.

2. Fan-out

Fan-out is the worker's alert-worker process. It receives a queue job with monitorId + incidentId + event type (down / up / cert_expiring / domain_expiring), looks up the routing — which channels should receive this alert for this monitor — and queues one delivery job per channel.

The routing chain is:

1. Monitor-level channels — if any are set in the Edit monitor dialog
2. Otherwise, all default channels — every row in Settings → Channels with isDefault=true

Dedup happens here too. Within 5 minutes, we won't fire the same alert type for the same incident on the same channel twice. Reminder alerts (cert expiring in N days) get a stronger dedup via a unique-key index on (kind, monitorId, trigger, expiresAt) so re-running the scheduler is always safe.

3. Delivery

Delivery is one process per channel type. Email goes through whatever EMAIL_PROVIDER is configured (console in dev, Resend Setup Day onwards). Slack and Discord both use webhook POST. Generic webhooks get a JSON envelope with the event type, monitor, and payload.

Each delivery is logged in the alerts table with status: pending → sent or failed. The Test button on the channels page hits the same delivery code path with a synthetic payload, so a passing test really does prove the production path works.

That's the whole pipeline

Three stages, three queues, one dedup key per stage.