#1081 Vulnerability monitoring follow-ups: escalation alerts, hardening and metrics Pull request opened by javuto Vulnerability monitoring follow-ups: escalation alerts, hardening and metrics Follow-up to #1076, #1078, #1079 and the NVD CPE change. It implements the items deferred from those reviews. Problem • Escalations never alerted. A finding that later became known-exploited, or rose in severity, never alerted again. So a "Known exploited only" rule missed the most common case: CISA adding a CVE the fleet already has. • Several edge cases from earlier reviews were open: • concurrent osctrl-tls sweeps; • out-of-order commits; • corrupt or stale cursors; • a rejected record blocking a feed; • stale posture passes; • mixed NVD settings across replicas; • no resume for a failed first NVD sync. • No metrics. Vulnerability monitoring exported no Prometheus metrics. • One frontend test was flaky. What changes Escalation alerts • Recording: osctrl-api records an escalation in a new

vuln_escalations

table whenever an open, confirmed finding becomes known-exploited or rises in severity. It stores the previous severity and KEV state. • Escalations are recorded in the same transaction that copies advisory flags to findings, and by the matcher. • Possible and resolved findings never escalate. • Rows are pruned after 30 days. • Alerting: the osctrl-tls watcher reads escalations through a second Redis cursor (

osctrl:alert:vuln:escalation-cursor

). • An escalation alerts only the rules it newly matches. A

kev

rule fires when CISA lists the CVE. A

high

rule fires when a medium finding becomes high, but not when a high finding becomes critical. • Details read "now known exploited" or "was medium". The entity gets a

:escalated

suffix, so the cooldown keeps it apart from the original alert. • Escalations have their own hourly budget and digest, separate from new findings. An escalation flood, as when NVD is first turned on, can't crowd out a new finding. Watcher hardening (osctrl-tls) • One sweeper at a time: a Redis lock lets only one replica sweep per interval. • Settling without clocks: a sweep takes only IDs the previous sweep already listed. A finding committed late by a slower transaction is therefore never passed by the cursor. Alerts arrive about a minute later. • Cursor lifetime: cursors are saved every sweep with a 7-day expiry. Re-enabling alerts after a long pause starts from now instead of replaying history. • Corrupt cursors reset to the newest record instead of stopping alerts. • Faster drain: the batch size goes from 2,000 to 20,000, so a large backlog drains about ten times faster. Alert rules • Pattern fields: vulnerability and node-state rules ignore pattern fields when compiling. A stray

match_type

no longer keeps a stored rule from loading. • Editing with the feature off: existing

vuln_finding

rules stay editable, so they can be disabled while

--vuln-enabled

is off. Only turning another rule into a vulnerability rule needs the feature. Posture • Category collision: posture records in categories the evidence source owns (

vulnerabilities

) are dropped. A posture query can no longer grade, or pass, the "Known vulnerabilities" control. • Which nodes are scored: a node without findings is evaluated when its last match assessed at least one package and its inventory is under 30 days old. • Clean nodes no longer lose the control after every advisory sync. • Nodes silent long enough that they're no longer re-matched stop passing. Feeds and storage (osctrl-api) • Migration: adding

vuln_node_state.assessed

re-matches every node once, so clean nodes keep their control right after upgrading. • Rejected records: a batch is stored in one transaction. Only if that fails is it retried one record per transaction. • A record the database rejects is skipped and counted. • If every record fails, the sync fails and keeps its cursor. • There are no per-record savepoints, which keeps Postgres from piling up subtransactions. • NVD key length:

vendor:product

keys longer than the 255-character column are dropped. • Resume: a failed full NVD sync resumes from its next page with its original start time (

vuln_sync_state.resume_index

/

resume_start

). • Mixed NVD settings: a replica with

--vuln-nvd-enabled

stamps

vuln_worker_state.nvd_active_at

and

nvd_active_by

(its host). • A replica with NVD off keeps NVD data, and logs a warning, while another host used NVD in the last hour. • On a single host, turning NVD off still drops the data on the next tick. Metrics With

--metrics-enabled

and

--vuln-enabled

, osctrl-tls's existing metrics server exports these gauges: •

osctrl_vuln_advisories

•

osctrl_vuln_feed_last_success_timestamp_seconds{source}

•

osctrl_vuln_findings_open{confidence}

•

osctrl_vuln_findings_resolved

•

osctrl_vuln_nodes_pending_match

•

osctrl_vuln_packages_not_assessed

They're read from the shared database and cached for a minute. A failed read exports nothing rather than zeros. API and frontend • Summary: the summary gains

kev_possible

, and the KEV card notes how many possible matches it left out. OpenAPI is regenerated. • Flaky test:

NodeConsolePage.test.tsx

mocked

useNavigate

with a new function on every render, so the console reopened its session on each re-render. • The mock is now stable, and a test asserts one session per console. • The osquery-prompt test had only passed because those re-renders wiped the scrollback. It now checks the command bar's prompt. Docs

pkg/vulns/README.md

(escalations, cursors, lock, settling, metrics, replica warning, resume),

ARCHITECTURE.md

(

vuln_escalations

) and the spec's implementation notes. Security and operational impact • With

--vuln-enabled

off, behaviour is unchanged, except for two alert-rule fixes that apply to all rules: • pattern fields are ignored for non-text sources; • existing vulnerability rules stay editable. • Schema (AutoMigrate, additive): • a new

vuln_escalations

table; •

vuln_node_state.assessed

triggers a one-time re-match; •

vuln_worker_state.nvd_active_at

and `nvd_active_by`; •

vuln_sync_state.resume_index

and

resume_start

. • New Redis keys: • `osctrlalertvuln:escalation-cursor`; •

osctrl:alert:vuln:sweep

(the lock); • budget keys now include a kind segment (

new

or

esc

). • Alert volume: turning NVD on can escalate many findings from

unknown

severity. The separate hourly budget bounds this to 10 advisories plus one summary per rule and environment per hour. • Metrics: every TLS replica exports the same fleet-wide numbers, so aggregate with

max()

, not

sum()

. • Known limitations: • A finding whose severity drops and later rises again alerts again. • An escalation recorded while its finding was resolved can alert if the finding reopens. • The sweep lock isn't extended for sweeps longer than 50 s. • The metrics queries have no timeout. • A finding that turns KEV before its first sweep can alert twice for a

kev

rule. Validation •

gofmt -l pkg cmd

(empty),

go vet ./...

,

go build ./...

• `go test ./... -count=1`: all 55 packages pass • `golangci-lint run ./pkg/... ./cmd/...`: 0 issues • `make openapi-check`: the diff adds

kev_possible

to

vulns.Summary

• No

go.mod

or

go.sum

changes • `cd frontend && npm run check && npm test`: 492 tests pass in 3 consecutive full runs • Live NVD check: the real API accepted the request format. A recent page and CVE-2023-4863 parsed correctly (CVSS 8.8, high, nine products with grouped ranges). • Mutation checks, each confirmed to fail the tests: • the escalation "newly matches" filter; • … jmpsec/osctrl