#30 Health reporting follow-ups: unknown state, start-up debounce, Retry-After, end-of-sync lag Pull request opened by javuto Health reporting follow-ups: unknown state, start-up debounce, Retry-After, end-of-sync lag Follow-up to #29. Four small fixes to the watcher and status report found in the final review of that change. No new routes and no change to the rollback chain. Problem After #29, osctrld reports osqueryd's state to osctrl. Edge cases made that report wrong or late: • False "down" on a failed check: if the end-of-sync poll failed, the state stayed unknown, but the report still said

osquery_running: false

, and nothing corrected it for up to a sync interval. • False "down" at start: the first observation after osctrld started was taken as truth. One miss at boot, before osqueryd was up, reported "down". Then it took 3 hits (~3 min) plus the 10-minute heartbeat window to clear. •

Retry-After

ignored: a 429 from osctrl-tls was retried on osctrld's own schedule whatever the header said. • Lag after a verification: the end-of-sync report used a single poll. After a flags or extensions change that worked, osctrl could see "down" for up to ~2 minutes, until the watcher's third hit. After a failed apply plus failed rollback it could see "up" for about a minute. What changes • Unknown is never reported. When the state is not known, nothing is sent. The next watcher tick or sync reports once a check succeeds. • Debounced start. The state starts unknown. • A first hit makes it up with no change reported and no heartbeat window. • A first miss stays unknown, and only a second consecutive miss makes it down (reported, heartbeating as before). • After the state is known, the rules are unchanged: down after 2 consecutive misses, up after 3 consecutive hits, and flapping never flips or reports. •

Retry-After

is honored. • On a 429 with a whole-seconds

Retry-After

, no report (heartbeat, state change or end-of-sync) is sent before that time, capped at 1 hour. • A missing, zero, negative, non-numeric or HTTP-date value means no extra delay. • The cap is applied before the multiplication, so a huge value cannot overflow or panic. • The 15 s send timeout and the 404/405 "not supported" handling are unchanged. • A verification is authoritative. • A flags or extensions verification completes after the required consecutive hits or after the 60 s timeout. Its result now goes straight into the health state and is reported at the end of the sync, in place of the single poll. • A cancelled wait is never used. • The result is cleared at the start of every sync, so a stale one can't leak into the next. • If both verifications run, the last one wins. • With no verification in the sync, the end-of-sync report observes by poll as before. •

down_since

is set on the transition to down and kept while down. •

README.md

documents the start-up rule,

Retry-After

, the end-of-sync report, and that a verification whose checks all fail times out and is reported as down, while a single failed poll is "no evidence". Security and operational impact • No new privileges, files or network calls. • Server influence: a server can delay osctrld's reports by at most 1 hour through

Retry-After

, which is only its own view of the host. • Restart behavior: no restart was added. osctrld still never restarts a dead osqueryd on its own. Known limits (deferred) • A watcher tick never reports unknown → up. After a host reboot osctrl can keep the previous "down" report until the next sync. osctrl stops counting a report after 15 minutes, and the node page marks it stale. • A

Retry-After

larger than int64 means no delay rather than the 1-hour cap. • After a verification times out,

down_since

is the end of the sync, so downtime is understated by up to ~60 s per verification. Validation • `golangci-lint run ./...`: 0 issues • `go test ./... -count=1`: passes (also with

-race

, run by the implementer) •

go vet ./...

for linux,

GOOS=windows

and `GOOS=darwin`;

gofmt -l cmd

clean • New tests, each seen failing first (one guard test passes both before and after): • Unknown state: a failed end-of-sync poll sends no report; a later successful observation does. • Debounced start: miss then hit leaves the state up with nothing reported; two misses report down once. • `Retry-After`: a 429 with

Retry-After: 120

holds reports back until the injected clock passes it; the cap, a huge value, a negative value and an HTTP date; an invalid or absent header adds no delay. • Verification feed: • after a verification that succeeded, the end-of-sync report says running immediately; • after one that timed out, it says down with

down_since

set; • a sync without a verification still polls, and a stale result is never reused. • Two existing tests changed: •

TestHealthWatcherUpNeedsConsecutiveHits

now starts with two misses, per the new start-up rule. •

TestSyncOnceFlagsThenExtensions

no longer expects the trailing poll after a verification, because that poll is replaced on purpose. • Both still check the same restart sequence and the 3-hit rule. jmpsec/osctrld