<#1005 Alerting system Stage 1: matching engine (`...
# osctrl
g
#1005 Alerting system Stage 1: matching engine (`pkg/alerts`) + `alertsEnabled` service flag Pull request opened by javuto Alerting system Stage 1: matching engine (
pkg/alerts
) +
alertsEnabled
service flag
Problem osctrl has no alerting: matches in status/result/query logs go unnoticed, and there is no notification layer. This PR lays the core engine — a high-performance rule matcher over the log ingest path — plus the service-config switch that keeps the whole subsystem inert until an operator opts in. Change New package
pkg/alerts
(8 files, ~1,700 lines incl. tests): •
models.go
— GORM models:
AlertRule
(unique name+env; sources: result/status/query logs and node inactive/recovered; substring/regex matching; field scoping; status-log severity floor; per-rule cooldown; channel refs),
AlertChannel
(logsinks-style typed config blob),
AlertHistory
(denormalized rule/channel names so deletions don't cascade). Tables:
alert_rules
,
alert_channels
,
alert_history
•
rules.go
— immutable
RuleSet
snapshots published through an `atomic.Pointer`: lock-free reads, atomic copy-on-write swap on refresh. DoS caps:
MaxPatternLen=512
,
MaxRulesPerEnv=50
•
matcher.go
— pure matchers (zero I/O): result logs (columns + snapshot rows), status logs (severity floor checked before any pattern work), distributed query results. Per-entry lowercased-value cache shared across all rules •
state.go
— Redis
SetNX
cooldown/dedupe gate keyed on rule+entity+detail-hash; fails open on Redis outage (alert availability beats perfect dedupe);
Release
for dispatch retry •
validate.go
— closed source set, regex/pattern validation,
ValidateRule
(mirrors
ValidateSink
) •
manager.go
— CRUD for rules/channels/history with sentinel errors, duplicate-key mapping, per-env rule cap;
LoadSnapshot
skips broken or disabled rules without failing the rest;
PruneHistory
for retention Service-config gating (follows the
postureEnabled
precedent exactly): •
pkg/config/types.go
— `AlertsEnabled bool \`yaml:"alertsEnabled"` `; seeds/resolves through the
service
section automatically (no registry change needed) •
pkg/config/flags.go
—
--alerts-enabled
/
SERVICE_ALERTS_ENABLED
, default false •
cmd/tls/main.go
— when disabled: alert tables are not created (no AutoMigrate), no snapshot loaded, ingest hook stays nil → zero cost on the hot path. When enabled: manager + rule store constructed, snapshot loaded at boot •
cmd/api/main.go
— same gate; alert API routes (Stage 4) will not register when off •
deploy/config/tls.yml
+
api.yml
— documented
alertsEnabled: false
•
docker-compose-dev.yml
— dev stack turns it on for Stage 2+ development • SPA
ServiceConfigPage.tsx
— help-text entries for the new field so it renders documented in the service-config UI Performance (priority) — benchmarked • 20 rules × 500-row result batch: 3.1 ms (design gate <5 ms) • 50 rules × 1,000 rows: 10.1 ms • No rules enabled: 2.1 ns/op, 0 allocations — matching is a no-op when the feature is off or no rules exist • Ingest path performs zero DB/Redis/lock operations: rules come from the in-memory snapshot, cooldown claims happen on worker goroutines only (Stage 2 wiring) Validation •
go build ./...
clean;
go vet
clean;
gofmt
clean on all touched files •
golangci-lint run
on all touched packages — 0 new issues (5 pre-existing in untouched files) • 21 tests: CRUD round-trips, rule cap, dup-key collisions, snapshot load skipping broken/disabled rules, severity floors, field-scoped missing-field contract, snapshot-row matching, failed-query matching, cooldown claim collapse/release, pattern-length/regex/source rejection, flag default-off + destination wiring (new
TestServiceAlertsEnabledFlagDefaultsOff
) • Redis claim tests auto-skip without
REDIS_URL
• Note: no tests existed for the touched area before this PR (new package); AutoMigrate of the three new tables is production-impacting but additive — tables only appear when the feature is first enabled Roadmap position Stage 1 of 7 (
alerts-engine
). Next: Stage 2 (
alerts-ingest-hook
) wires the buffered worker + `ProcessLogs`/`ProcessLogQueryResult` taps in osctrl-tls behind this flag; Stage 4 registers the management API; nothing operator-visible ships until then. jmpsec/osctrl