#1100 osctrld health reports: node page banner and alertable metrics Pull request opened by javuto osctrld health reports: node page banner and alertable metrics Follow-up to #1096, #1097 and #1098. Paired with the osctrld change that sends the reports (rollback chain, watcher,

osctrld-status

sender). Problem When osqueryd goes down on a host, osctrl sees a silent node and no reason. osctrld only reported flag and extension apply results on its hourly sync, so nobody could tell that osquery was down or why, and nothing could be alerted on. What changes New osctrl-tls route •

POST /{env}/osctrld-status

(body ≤ 4 KiB, environment secret in the body). It is registered inside the existing osctrld block behind

osctrld.enabled

, wrapped by

OsctrldMiddleware

like the other osctrld routes. That keeps the #1097 rate limits, failed-secret blocking, audit cap and latency metrics. It uses the shared secret helper. • The report carries:

osquery_running

,

down_since

(omitted when running), and a rollback

stage

plus

event_id

for flags and extensions. • Responses: 403 for a wrong secret, like the other routes. 400 for malformed JSON, a bad uuid, or a uuid that is not a node of that environment. It never answers 404, because osctrld reads 404 as "route not served". A node of another environment and an unknown node get the same empty 400, so nothing leaks. 200

{}

on success. • Stages are validated (flags: `none|environment|last_known_good|exhausted`; extensions:

none|restored|exhausted

) and event ids must match

^[0-9]+:[0-9a-f]{64}$

. An invalid kind is dropped and the rest of the report is still stored. Storage (

pkg/nodeconfig

) • New columns on `node_osctrld_status`: running state,

down_since

,

reported_at

(indexed), flags and extensions stage, flags and extensions last event id. The migration is additive; string columns have

default:''

. •

RecordReport

upserts the state, then counts a rollback event as new only if a single compare-and-set

UPDATE … WHERE <kind>_event_id IS NULL OR <kind>_event_id <> ?

changes a row. Heartbeats repeating the same event don't re-count or re-audit, and two replicas can't both count it. The NULL-safe predicate matters for rows that existed before the migration. •

RecordFetch

is untouched and doesn't clobber report fields, and the other way round. •

down_since

values from the node clock outside years 1970–9999 are ignored. Audit and metrics • For each new rollback event, one audit entry via the new

AuditLogManager.RolledBackOsctrld

(warning, node log type). •

osctrld_rollbacks_total{environment,kind,stage}

counter, incremented once per new event. Labels only use the environment name, a fixed kind, and a validated stage. •

osctrld_osquery_down{environment}

gauge: nodes whose latest report says osqueryd is down and is no older than 15 minutes. It emits a series for every environment, 0 when none are down, with no per-host labels. The 15 minutes is one constant,

nodeconfig.ReportFreshness

, shared with the node page. • The gauge is a scrape-time collector built like the vulns collector: a 15 s cache, a 5 s read timeout and a 1-minute backoff after a failure, and it recovers from panics. A slow database can't stall

/metrics

. On a failed read it exports nothing for that scrape. API and node page • The node flags view (

osctrld

object) gains

osquery_running

,

down_since

,

reported_at

,

report_stale

(true when the report is older than 15 minutes),

flags_stage

and

extensions_stage

, all omitted when absent.

down_since

is clamped so it is never later than

reported_at

. •

last_fetched_at

is now a pointer and is omitted, instead of

0001-01-01

, when a report created the row before any flags fetch. Rows that fetched keep the same JSON. • Swagger comments say the new fields are host-reported and untrusted;

osctrl-api.yaml

is regenerated. • The new

OsctrldDownBanner

shows once, at the top of the node page on every tab, when osctrld says osquery is down. It states the time reported, the rollback stage in words, the last apply error, and "may be stale" when the report is old. It is built from the existing node-flags query, so there is no extra request, and host text is rendered as plain text. It is removed from the Config › Flags view so it never shows twice. •

ARCHITECTURE.md

documents the route and metrics. Security and operational impact • New input: this adds one more authenticated, rate-limited, size-limited route on the externally exposed service. A holder of the environment secret can report for any node of that environment, the same trust as requesting that node's flags today. • Alerting:

osctrld_osquery_down

is derived from the database, so every osctrl-tls replica exports the same value. Aggregate with

max by (environment)

, never

sum

.

osctrld_rollbacks_total

is counted by one replica only, so

sum

is correct. If the database read fails the gauge disappears for that scrape, so pair alerts with

absent()

if outages must not resolve them. • Compatibility: a released osctrl-tls answers 405 (not 404) to this route because it registers

GET /

. The paired osctrld treats 404 and 405 as "not supported". An old osctrld never reports; its nodes show no report fields and count as 0. • Cost: the gauge does two small DB reads per scrape at most every 15 s per replica. Known limits (deferred) • If

RecordReport

fails partway when both kinds carry a new event, the first event's audit entry and counter increment are lost on retry. • A database error during node lookup is answered 400, like the existing flags path. • Rollback audit rows have no per-environment cap; only holders of the environment secret can reach them, and the per-address limits still apply. • The OpenAPI description of the new view fields only attaches to

osquery_running

. • A report-only row stores a zero

last_fetched_at

. I could not check this against MySQL strict mode. Validation • `go test ./... -count=1`: 57 packages pass •

go vet ./...

,

gofmt -l pkg cmd

(empty), `golangci-lint run ./...`: 0 issues • `make openapi-check`: clean • Frontend: 568 unit tests pass and

tsc

is clean • New tests, each seen failing first: • Storage: • state stored; a repeated event isn't new; two managers count an event once; • invalid stage or event id is dropped; • a report can create the row;

RecordFetch

keeps report fields; • the first event on a row with NULL columns counts; • an absurd

down_since

is ignored. • Down count: stale and running hosts aren't counted; grouped by environment. • Route: • the wire-contract JSON is stored; another environment's node, a bad uuid or body, and a bogus stage store nothing; • an event is counted and audited once across three identical reports; • the route returns 429 when the address is blocked. • Collector: the gauge per environment (0 for the rest); survives a DB error; a stalled DB read doesn't block the scrape; the TTL cache; the backoff. • API view: report included; stale marked;

down_since

clamped; old shape kept without a report; no zero

last_fetched_at

. • Frontend: • the banner states, stale text, plain-text rendering, and absence when running; • once on the node page's default tab; • the flags view without the banner and with a missing

last_fetched_at

. jmpsec/osctrl