#1109 Gradual rollout for the environment osquery target Pull request opened by javuto Gradual rollout for the environment osquery target Problem Setting an environment osquery target today is applied by every host at its next sync, within about an hour, with no throttle. A bad version reaches the whole environment before anyone sees the first rollback. The per-host pin is the only way to test first. Change Setting the environment target now starts a staged rollout: • Stages: 1% → 10% → 50% → 100%, about 2 hours per stage. The stages are fixed and not configurable. • Widening: stages open by time only. The stage is computed from the stored row and the clock, so there is no background job and every replica agrees. • Pause: the first rollback of this rollout's target pauses it. A paused rollout also stops hosts that have not upgraded yet, including hosts in the open stage. Hosts already mid-upgrade finish. • Resume and Abort: • Resume continues from the stage where it stopped and restarts that stage's wait. • Abort stops for good and clears the environment target. • Hosts that already upgraded stay as they are, and targets never downgrade. • Both are environment-admin only, audited, and return 409 when the state does not allow them. • Who is in a stage: a stable hash of (environment id, target version, node uuid). The four thresholds are computed once at rollout start from the environment's hosts, so stage 1 always has at least one host. Hosts enrolled later use the same thresholds. • Pins: a pin ignores the rollout and still upgrades its host immediately. Hosts already at or above the target are never affected. Pieces: • Rollout state in `pkg/osqueryversions`: new

osquery_rollouts

table keyed by (environment, request id). Setting the target creates a new rollout and replaces the old one. Clearing the target aborts it. Progress counts (eligible, upgraded, waiting, rolled back) and the rolled-back host list are computed on demand with no new schema for them. • `Resolve`: new reasons

rollout_pending

and

rollout_paused

for environment-sourced targets, evaluated after

ahead_of_target

. Both carry

source

and

request_id

and no target. The route stays O(1) and never scans hosts. • osctrl-tls: • A valid rollback carrying the rollout's request id pauses the rollout with a compare-and-set. • The pause is audited once and names the host. • A failed pause write answers 500 so osctrld retries. • A rollback pauses only if it was first recorded, by server time, after the last Resume. osctrld resends its last rejected result, and without this rule a Resume would be undone about one sync later. • New metric

osctrld_osquery_rollout_paused{environment}

. Aggregate it with

max by (environment)

. • osctrl-api: • The environment osquery response gains a

rollout

object (null when there is none). • New endpoints

POST /api/v1/environments/osquery/{env}/rollout/resume

and

.../rollout/abort

. • OpenAPI is regenerated. • Frontend: a rollout panel on Environment › Osquery. • Running and paused states show the stage bar, counts, time to the next stage and the rolled-back hosts. • Resume and Abort sit behind confirmations. • Completed and aborted show one line. • The Set target confirmation says the rollout starts at the first stage. • The node page has wording for the two new reasons. Behavior notes • A rollback after the rollout reaches 100% still pauses it, so the 50% → 100% wave can be stopped. Resume and Abort both work from that state. This differs from the spec's "no gating after completion", and the spec carries an amendment. • Hosts that rolled back are not retried under the same request id after Resume. This is osctrld's existing rejected-memory rule. Other hosts continue. • Targets set before this change have no rollout row and are never gated. • The node page shows a generic "not in the rollout yet" sentence. It does not give the stage or time, because the node response has no stage information. Security and operational impact • Permissions: Resume and Abort check the environment-admin permission server-side. Denials are audited, and cookie sessions need a CSRF token on both POST routes. • Untrusted input: • Host-reported values on the pause path are validated against the existing reason allowlist, a version format and a 32-hex request id. • Host names go through a cleaner that drops control characters and caps the length. • The UI renders them as plain text. • A node can only pause the rollout of its own environment. • Schema (additive only): • new table

osquery_rollouts

• new column

upgrade_event_at

on the node status table •

resumed_at

on the rollout • Cost: each GET or mutation on the environment osquery endpoint loads the environment's non-Windows hosts to compute counts. This is fine at normal sizes. Caching is the obvious follow-up for environments around 100,000 hosts. Aborted rollouts skip the scan. • Rolling deploys: a mixed-version window can record an upgrade event without

upgrade_event_at

. That event does not pause a rollout. Nothing is lost, because old replicas have no pause logic. • Windows: hosts are excluded from the rollout, matching the existing

unsupported_platform

answer. Validation •

go test ./... -count=1

,

go vet ./...

,

gofmt -l cmd pkg

• `golangci-lint run ./...`: 0 issues • `make openapi-check`: clean • Frontend:

npm test

(638 tests) and

npm run check

• New tests cover: • thresholds for tiny environments • the effective stage over time • concurrent pause • the Resume-then-resend case • pins and

ahead_of_target

• cross-environment isolation • permission denials • the Abort and Resume state rules • host-name cleaning • Not done: a manual rollout on a real staging environment. Please check that stage 1 picks hosts, stages open on schedule, a forced rollback pauses, Resume continues and Abort clears the target. Compatibility Pairs with the osctrld change that records the two new reasons. Merge order does not matter: older osctrld releases ignore the reasons, so gated hosts simply do not upgrade, and nothing breaks. jmpsec/osctrl