#29 Keep reporting when osqueryd is down Pull request opened by javuto Keep reporting when osqueryd is down: verified rollback chain, health watcher, status report Follow-up to #27 and #28. Paired with the osctrl change that adds the

osctrld-status

route, the node page banner and the metrics. Problem osctrld restarts osqueryd after changing flags or extensions, and rolls back when osqueryd doesn't survive. Gaps found in

flagsguard.go

and `service.go`: • Unverified rollback: the restart after a flags rollback was never checked. If the environment flags also failed, osctrld only logged it and stopped. A flags rollback never restarted onto anything but the environment flags. • Unverified extensions rollback: the same gap existed for the restart after an extensions rollback. • Silent when down: osctrld only talked to osctrl on its hourly sync, and only about flag and extension apply results. When osqueryd was down, osctrl saw a silent node with no reason, and nothing could be alerted on. What changes Rollback chain for flags • A failed flags check now falls back through: attempted flags → environment flags → last known good flags → stop. Each step is written, restarted and verified the same way as a normal apply. • The last known good flags are saved whenever a flags check passes. They live in

<flagfile>.osctrld-lastgood

, written atomically with mode 0600 and checked against a stored SHA-256. A torn or tampered file counts as missing. The file is in

protectedFiles()

, so an autoload path can't target it. • Each failed content (by SHA-256) is remembered and not retried while osctrl keeps serving it. A chain restarts osqueryd at most once per candidate, and a content that already failed is never restarted onto again. • When no step keeps osqueryd running, osctrld stops with stage

exhausted

, leaves the flagfile on the content the next sync will settle on, and keeps syncing. Flags the operator fixes in osctrl are applied normally on the next sync. • A chain interrupted by osctrld stopping or the host rebooting resumes where it stopped. The original attempted flags stay the reported result, and the rollback event id stays the same. Extensions • The restart after an extensions rollback is now verified. The result is recorded as a rollback event with stage

restored

or

exhausted

. Otherwise extensions behave as before. Health watcher and status report • osctrld polls osqueryd every 60 s. It calls it down after 2 consecutive misses and up after 3 consecutive hits, so a service manager respawning a crashing osqueryd doesn't flip the state. • It reports to osctrl-tls (

POST /{env}/osctrld-status

) in these cases: • immediately on every state change; • about every 5 minutes (jittered) while osqueryd is down, and until it has been up for 10 minutes; • at the end of every sync. • The report carries: whether osqueryd is running, since when, and the rollback stage plus event id for flags and extensions. • Each report has a 15 s timeout of its own (not the 10-minute download timeout), so an unreachable or stalled osctrl can't stall the service loop. • osctrld never restarts a dead osqueryd on its own. A sync still runs and applies fixes while osqueryd is down. • Old osctrl: a 404 or 405 answer means "route not served". A released osctrl-tls answers 405 because it registers

GET /

. osctrld pauses reports until the next sync and logs once at debug level instead of warning every interval. State and compatibility • New fields in the flags state file and a new

rollback

field in the extensions state file are all optional. Old state files load cleanly. •

writeStateFile

is now atomic (temp file plus rename). This also covers the extensions state: the rename replaces a symlink instead of following it as root, but a state file's owner and inode change on each write. •

README.md

documents the chain, the watcher, the report cadence and the new files. The link to the git-ignored

ARCHITECTURE.md

is removed. Security and operational impact • No new privileges: osctrld sends one more small JSON POST per sync, plus a heartbeat every ~5 minutes while osqueryd is down, to the osctrl it already talks to. It uses the same environment secret, which is only ever in the request body and never logged. • Last-good file: it holds the flags osctrld already runs as root. It is 0600, written by rename, and refused as an autoload target. • Restart behavior: a bad flags push now costs at most two extra verified restarts per distinct served content, then osctrld stops. It never restarts osqueryd without a changed input. • Known limits: • osctrl can see "down" for up to about 2 minutes after a fix that worked, because the end-of-sync report uses a single poll. • A first miss right at osctrld start reports "down" until 3 hits. • A 405 from a misconfigured proxy would pause reports with only a debug log. •

Retry-After

is not read. •

writeFileAtomic

does not fsync. Validation • `golangci-lint run ./...`: 0 issues • `go test ./... -count=1`: passes •

go vet ./...

for linux,

GOOS=windows

and `GOOS=darwin`;

gofmt -l cmd

clean • Each new test was seen failing first: • Flags chain: • environment then last good; exhausted without a loop; • environment flags themselves bad; no last good yet; • last good recorded on ok; a last good with a wrong hash is ignored; • a new failed input after exhausted doesn't restart onto already-failed content; • a fixed flags push is applied while exhausted; • an interrupted chain resumes (table: runs, fails then last good runs, nothing runs) and keeps the event id; • last-good file is protected from autoload. • Extensions: the rollback restart is verified;

restored

vs `exhausted`; a clean check leaves no rollback event; no second restart. • Report: • the JSON matches the wire contract byte for byte; • stages map correctly, with

"none"

and no event id when there is no rollback; • 404 and 405 are "not supported" and not logged as errors; 429 is a real error; • a stalled server times out within the bound. • Watcher: • down needs 2 misses and up needs 3 hits, and flapping doesn't report; • heartbeat cadence and the 10-minute stable window; • a state change reports immediately and the end of a sync reports; • the loop survives an unreachable osctrl and keeps applying fixes while down. • Three existing

syncOnce

tests were updated for the extra end-of-sync poll. They still assert the restart sequence and count. jmpsec/osctrld