#1007 Alerting system Stage 3: notification channels (webhook + email) and dispatch fan-out Pull request opened by javuto Alerting system Stage 3: notification channels (webhook + email) and dispatch fan-out Problem Stages 1–2 delivered rule matching on the ingest path and an async dispatch pipeline, but the pipeline had nowhere to send: hits reached the dispatch worker and only landed in

alert_history

. This PR adds the notification layer — webhook and email channel types with a pluggable registry, a dispatcher with failure isolation, and the cache/refresh wiring to make channel edits take effect without a restart. Change Channel registry — new

pkg/alerts/channels.go

• Pluggable

ChannelRegistry

following the exact log-sinks pattern:

ChannelSpec

with typed

FieldSpec

form schemas (the SPA will render channel forms generically),

SecretFields

for API redaction, and generic

decodeTyped[T]

config decoding •

ValidateChannelType

/

ValidateChannelConfig

mirror `ValidateSink`; manager

CreateChannel

/

UpdateChannel

now reject bad types and malformed configs before rows are stored Webhook channel — new

pkg/alerts/channel_webhook.go

• POSTs the alert as JSON (rule, environment, node, entity, detail, timestamp) • Optional HMAC-SHA256 signing (

X-Osctrl-Signature

) so receivers can authenticate the payload • SSRF guard: loopback, link-local, and RFC1918 targets are refused by default; unresolvable hosts are refused too (fail-closed).

allowPrivateTargets

opts in for the dev stack's sink-catchall container • Retries: 3 attempts with capped exponential backoff; 4xx responses (other than 429) short-circuit — a definitive rejection is not retried Email channel — new

pkg/alerts/channel_email.go

• Per-channel SMTP relay config (host/port/credentials/from/to); port 465 = implicit TLS, STARTTLS upgrade elsewhere (default on) • RFC-5322 message rendering (subject with rule + env, body with node/entity/detail/timestamp); delivery abstracted behind a

send

seam so tests validate rendering and recipients without a relay Dispatcher — new

pkg/alerts/dispatch.go

• Implements the Stage-2 `DispatchSink`: resolves hit channel IDs →

alert_channels

rows → memoized senders (cache keyed on the config blob; `RefreshChannel`/`Reset` invalidate) • Failure isolation: a dead webhook does not block the email; each successful delivery writes its own

alert_history

row so the audit trail shows exactly who was notified • Error semantics refined: disabled, unknown, or dangling channels are skipped without counting as failures (an operator disabling a channel is intent, not an outage); zero usable channels is a no-op success; an error is returned only when real deliveries were attempted and all failed — which drives the worker's claim-release retry path correctly Worker —

pkg/alerts/worker.go

•

sinkSelfRecords

flag (auto-detected for

*Dispatcher

): the channel dispatcher writes per-channel history itself, so the worker skips its generic catch-all row to avoid double-recording Wiring —

cmd/tls/main.go

• Dispatcher constructed as the worker sink when

--alerts-enabled

is set •

watchAlertRules

now resets the channel sender cache alongside the rule snapshot, so channel config edits rebuild senders on the same 5-minute refresh (Stage 4's service-command trigger will make it instant) Validation •

go build ./...

,

go vet

,

gofmt

— clean •

golangci-lint run ./pkg/alerts/... ./cmd/tls/...

— 0 issues • 19 new tests, all passing, plus the full Stage-1/2 suite: • webhook: deliver + HMAC verify round-trip against a live test server, no-secret/no-header, retry count on 500, no-retry on 403, SSRF rejection matrix (loopback/localhost/private/link-local/unresolvable, and the allowPrivateTargets opt-in), bad config rejection • email: render + recipient parsing via the delivery seam, config validation errors • registry: completeness, type normalization, config decode validation • dispatcher: fan-out isolation (good + dead channel → only the good channel in history), all-deliveries-fail error, disabled/unknown/ dangling channel skip, sender-cache invalidation on config edit • Pre-existing suites (

pkg/logging

,

cmd/tls

,

cmd/tls/handlers

) green Security notes • Webhook SSRF posture reviewed against the repo's security lens: private and loopback targets are denied unless a deployment explicitly opts in, and unresolvable hosts fail closed • Channel secrets (

secret

for webhooks,

password

for SMTP) are flagged in the registry's

SecretFields

so the Stage-4 API can redact them in read responses, the same way log-sink credentials are handled Roadmap position Stage 3 of 7 (

alerts-channels

). The pipeline is now complete end-to-end ingest → match → cooldown → deliver → per-channel history, observable via Prometheus. Next: Stage 4 (

alerts-api

) — CRUD handlers and routes in osctrl-api, CLI

alert

commands, OpenAPI annotations, and the service-command trigger that turns the 5-minute refresh into an instant one. jmpsec/osctrl