#31 osquery upgrades: verified install, config-preserving restart, automatic rollback Pull request opened by javuto osquery upgrades: verified install, config-preserving restart, automatic rollback Follow-up to #27, #28 and #29. Paired with the osctrl change that adds the version library, the per-environment target, the per-host pin, the

osctrld-osquery

route and the node page. Problem osctrld installs osquery once, at install time, and takes the version from one server-wide value with one digest. That digest cannot fit osquery's different package per format and architecture, and nothing moves a running fleet to a new osquery version. What changes New service-loop step: osquery upgrade • osctrld asks osctrl for its target (

POST /{env}/osctrld-osquery

) at the end of a sync, sending its host UUID,

GOOS

,

GOARCH

, package format and installed version. The secret is only in the body. • It only upgrades when all of these hold, with at most one attempt per sync: • the host is not Windows; • no upgrade is already pending; • nothing else changed in this sync (flags, certificate, extensions), and no osqueryd verification ran; • the health state is known and osqueryd is running, confirmed by a fresh check that never feeds the health watcher. • A pending upgrade is resumed at the start of every sync, before flags, certificate and extensions are written. • The resolution from osctrl is validated before anything happens: both package versions, both SHA-256s (64 lowercase hex) and the request id (32 hex). Anything else is `invalid_resolution`: it is logged and never reported. The upgrade itself 1. Download the target and the previous version from the official osquery server, using the per-version URLs. Upgrades never use the install-time

--osquery-package

override, and they verify the server's certificate even with

--insecure

. 2. Verify each SHA-256 while streaming. A package is never held in memory, and the download is capped at 512 MiB. Then run the platform signature check where one exists. 3. Copy both packages into a root-only directory (

<flagfile>.osctrld-upgrade

, 0700, refused if it is a symlink or owned by someone else) and re-hash them. Installs only ever run from those copies, and the hash is checked again right before each install. 4. Snapshot the flagfile, secret file, certificate and the autoload file (contents, mode and owner), and record a listing of the extensions directory. The autoload path comes from flags that environment admins edit, so it is only snapshotted if it passes the existing

checkAutoloadPath

. Save the snapshot and the pending marker before touching the host. 5. Stop osquery, install the target, stop it again (a package may start it) and restore the snapshot. Then compare the extensions listing and the installed version, and only then start osquery. osquery never runs with packaged defaults between the install and the restore. 6. Wait for osqueryd, then confirm with osctrl, within 5 minutes, that it sees the node reporting the target version. The check polls every 30 seconds, is a plain string comparison with no clock comparison, and honors a 429

Retry-After

. 7. On success, record

ok

, clear the rejected memory and remove the stashed packages. Rollback • If the install fails with the version changed or unreadable, osqueryd doesn't start, the extensions directory changed, or the new version never reports, osctrld: • installs the previous package (re-verified first, with

rpm -Uvh --oldpackage

for rpm); • restores the snapshot, restarts osquery, and records

rejected

with the reason and an event id; • remembers the target by version, digest and request id, and does not retry it until one of them changes. An admin re-applying the target in osctrl produces a new request id. • If the install fails but the installed version is still the previous one (for example a dpkg or rpm lock held by unattended-upgrades), nothing is rolled back and no rollback is counted. osctrld restores the snapshot, starts osquery, and records

skipped

with

install_failed

. • Skips for

target_unavailable

,

previous_unavailable

,

digest_mismatch

and

install_failed

are remembered for 6 hours per target version, digest and request id, so a wrong digest in the library does not re-download packages on every sync. • An osctrld stopped mid-upgrade resumes, including a rollback cut short by a cancel (the reason is saved before the rollback starts). Reporting • The existing status report gains an optional

upgrade

object (

result

ok|rejected|skipped,

version

,

reason

,

request_id

,

event_id

only for

rejected

), built from the last recorded result. Only reasons in the contract's allowed set are sent. • A

skipped

result never replaces a rejection that has not been reported yet. A later healthy sync restores a reported rejection as the visible result, so the node page never shows a stale "osqueryd not running". • The upgrade step's own osqueryd wait feeds the end-of-sync report, so osctrl sees the real state in the same sync. Supporting changes •

downloadPackage

streams to disk (it used to read the whole package into memory), with the size cap above and no file left behind on any failure. •

installPackageAs(format, path, downgrade)

is new, and

installPackage

calls it without downgrade.

msi

refuses a downgrade without running any command. • Owners are recorded and restored on POSIX.

snapshot.json

is 0600 inside the 0700 directory and is deleted when the upgrade ends. New state:

<flagfile>.osctrld-upgrade-state.json

, added to

protectedFiles()

. • The README has a new "osquery upgrades" section. Security and operational impact • osctrld never installs an unverified package: digests come only from osctrl, are re-verified from the stashed copy immediately before install, and

allowUnverified

is never set. • An upgrade starts only with osqueryd already running, and osctrld never restarts a dead osqueryd on its own. The upgrade's own stops and starts are the only exception, and they are inside the upgrade or its resume. • The secret is only in request bodies. The snapshot file holds the secret file's contents, in the root-only directory, and is deleted afterwards. • A bad library digest costs at most one cheap refusal per host every 6 hours. • Windows hosts never upgrade, and the route answers

unsupported_platform

for them. • Needs a real-host test before release: a deb, rpm and macOS pkg upgrade and rollback, especially whether a package's install scripts start osquery or prompt about config files. I can't run real package managers in tests. Known limits (deferred) • Temporary downloads in

/tmp

can be left behind if osctrld is killed during a download. • A crash loop inside the 5-minute confirmation window never converges to a rollback. • Files a package creates that did not exist before are not removed on rollback, and extension files a package adds are detected but not removed. • A development build whose

--version

is not a plain version number gets a warning on each sync and never upgrades. • An osctrl outage during a rollout rolls back every host that is confirming at that time. This is intended. • The

install

command still downloads with

--insecure

when set (unchanged). Validation • `golangci-lint run ./...`: 0 issues • `go test ./... -count=1`: passes (also with

-race

on the upgrade, sync, health and status tests) •

go vet ./...

for linux,

GOOS=windows

and `GOOS=darwin`;

gofmt -l cmd

clean • New tests, each seen failing first. Deliberate breakages were also checked: saving the marker after the install, starting osquery before the restore, dropping the autoload check, disabling the skip memory, and ignoring the recorded owner each failed the expected tests. • Packages: • streaming with a chunked body, digest mismatch and size cap (every failure leaves no file); • rpm

--oldpackage

, … jmpsec/osctrld