#1082 Vulnerability monitoring: close the deferred items Pull request opened by javuto Vulnerability monitoring: close the deferred items Follow-up to #1081. It implements the items deferred in that review. Problem • Repeat alerts: a finding that dropped in severity and came back re-alerted as an escalation. This happens when NVD is off for a while or the KEV feed comes back truncated. • Old escalations on reopen: an escalation recorded before a finding was resolved could alert when the finding reopened. • Double KEV alert: a finding that turned KEV before the watcher first saw it alerted twice on a

kev

rule. • Sweep lock: • The lock wasn't extended, so a sweep longer than 50 s could overlap another replica. • Sweeps had no deadline. • Metrics: • Readings had no timeout. • Feeds the fleet no longer uses kept exporting a frozen last-sync time and marked every view stale. • Every TLS replica exports the same fleet-wide numbers, and the docs didn't say how to aggregate them. • Migration: a failed column check re-matched every node on each boot. What changes Escalations (osctrl-api) • Peak tracking: findings gain

peak_severity

and

ever_kev

, the highest state each finding ever reached. • An escalation is recorded only when a finding goes above its peak. •

refreshFlags

raises peaks after copying advisory flags, and the matcher keeps them too. • The new columns default to

''

and

false

. The backfill also matches NULL, so findings that existed before the upgrade are covered. • Reopens: findings gain

reopened_at

, stamped when a resolved finding reopens.

EscalationsAfter

hides escalations recorded before the last reopen and keeps those recorded after it. Watcher (osctrl-tls) • Unalerted findings: escalations are skipped for findings the watcher hasn't alerted yet. That finding alerts once, with its current state. • The check uses the findings cursor from before the previous sweep, because settling takes an escalation one sweep after it's listed. • Trade-off: an escalation committed in the milliseconds between one sweep's two queries can be skipped. The finding itself has still alerted. • Lock: the sweep lock is held for the whole sweep and released by token compare-and-delete (a Lua script). It has a 5-minute TTL in case the holder crashes. • Deadline: each sweep gets a deadline of the lock TTL minus 30 s, so two replicas never sweep at the same time. Feeds and metrics • Unused feeds: sync rows of OSV feeds that no node reports are dropped each sync. • Never because of `--vuln-ecosystems`: an ecosystem the allow-list leaves out but nodes still run keeps its row, so its findings aren't resolved as unassessed. • Never when the fleet is empty, so re-adding a node doesn't re-download archives. • Metrics timeout: each metrics reading has a 10 s deadline. A failed or slow reading exports nothing, never zeros. • Docs: they now say to aggregate the gauges with

max()

, since every TLS replica exports the same fleet-wide values. Migration • Column check: if the check for the new column errors,

Migrate

now returns the error instead of treating the column as missing. Security and operational impact • Flags off: no behaviour change with

--vuln-enabled

off. • Schema (AutoMigrate, additive, with defaults):

vuln_findings.peak_severity

,

vuln_findings.ever_kev

and

vuln_findings.reopened_at

. • Upgrade: no burst of escalations. Rows from before the upgrade take their current values as the peak. • Redis: the sweep-lock value is now a per-holder token. • If the holder crashes, sweeping pauses for up to 5 minutes; this was 50 s. • A lost

SETNX

reply can also block sweeping for up to 5 minutes. Accepted. • Known limitations: • A failed metrics reading isn't cached, so each scrape against a stalled database waits up to 10 s. • The live-Redis tests for the lock and the alert budget skip without

REDIS_URL

. Validation •

gofmt -l pkg cmd

(empty),

go vet ./...

,

go build ./...

• `go test ./... -count=1`: all 55 packages pass.

go test ./pkg/alerts -race

passes. • `golangci-lint run ./pkg/... ./cmd/...`: 0 issues • `make openapi-check`: no changes, since the new fields are not serialized • No

go.mod

or

go.sum

changes • `cd frontend && npm test`: 492 tests pass • New tests, each RED before its fix: • escalation only above the peak (

refreshFlags

and the matcher); • peak raise; • NULL backfill on upgraded rows; • escalations before and after a reopen; • skipping escalations of unseen findings, and of a finding alerted the sweep before; • lock release; • dropping unused feeds while keeping allow-listed and empty-fleet ones; • metrics failing closed. • Mutation checks: the peak raise, the matcher peak, the reopen filter and the drop guards each fail their tests when removed. • Fresh-context review. Its Important findings are fixed, each with a test that failed first: • NULL in the new columns on upgraded rows; • feed drops that resolved real findings; • the residual double KEV alert. jmpsec/osctrl