#1102 osquery upgrades: version library, environment target, per-host pin, and upgrade reporting Pull request opened by javuto osquery upgrades: version library, environment target, per-host pin, and upgrade reporting Follow-up to #1096, #1097, #1098, #1100 and the health follow-ups. Paired with the osctrld change that performs the upgrade (verified install, config-preserving restart, automatic rollback). Problem The osquery version and digest osctrl hands to osctrld are one server-wide value with a single digest, which cannot fit osquery's different package per format and architecture. Nothing lets an operator move a running fleet to a new osquery version, say which hosts are behind, or see why a host did not upgrade. What changes Version library, targets and pins (new

pkg/osqueryversions

) • Library: an admin-entered SHA-256 per `(version, platform, arch, format)`: linux deb and rpm on amd64 and arm64, and the macOS pkg (stored as arch

universal

). Windows/msi is rejected. • Targets: a target version per environment and an optional per-host pin (optional expiry, default none). Setting or re-setting either always generates a new opaque

request_id

, even for the same version. osctrld uses that id to decide whether to retry a target it rejected. • Guards: a version in use by a target or an unexpired pin cannot be deleted, and its existing digests cannot be changed. New keys can still be added. •

Resolve

gives osctrld one answer per host: the pin if unexpired, else the environment target. The reasons, in order, are

unsupported_platform

,

no_target

,

ahead_of_target

,

no_digest_for_target

and

no_digest_for_installed

. •

ahead_of_target

is checked before the digest reasons, so an up-to-date host never reports a missing digest. • A host whose running version has no digest in the library is not upgraded, because there would be no verified way back. The node page says which digest to add. • Version strings are validated everywhere: up to six digits per part, two to four parts, overflow-safe comparison, and a trailing newline is rejected. osctrl-tls route

POST /{env}/osctrld-osquery

• It is registered in the osctrld block, only when osctrld is enabled, through the same

OsctrldMiddleware

and secret check as the other osctrld routes. The #1097 rate limits, failed-secret blocking and audit cap apply. • The request carries the host UUID, platform, arch, format and installed version, and the response is exactly the wire contract.

reported_version

is the node's stored

osquery_version

, which osctrld uses to confirm an upgrade by string equality. • A node of another environment and an unknown node get the identical empty 400. It never answers 404 or 405, because osctrld reads those as "route not served". Only

gorm.ErrRecordNotFound

is a 400, and anything else is a logged 500. • Windows: an unsupported platform is a 200 with

unsupported_platform

, and the node must still belong to the environment. Upgrade results from osctrld • The existing

osctrld-status

report gains an optional

upgrade

object (

result

ok|rejected|skipped,

version

,

reason

,

request_id

,

event_id

). Everything is validated before storage: a closed result set, a closed reason set, valid versions, a 32-hex request id, and an event id (

<unixnano>:<64 hex>

) required for

rejected

only. • An invalid upgrade is dropped without touching the rest of the report. • The new columns are additive, default to empty strings, and the NULL-safe compare-and-set also works for rows that existed before the migration. • Rollback events: a rejected result is counted and audited once per event id, across replicas, through the existing path. This includes the case where

RecordReport

fails partway. • New counter

osctrld_osquery_upgrade_rollbacks_total{environment}

. • New audit entry

RolledBackOsquery

. Metric

osctrld_osquery_behind{environment}

• It counts hosts with a fresh osctrld report (the shared 15-minute window) whose stored version is lower than their effective target. It is a scrape-time collector with the same 15 s cache, 5 s timeout, 1-minute backoff and panic recovery as the down gauge. • The join is on UUID and environment, so a host that moved environments is not counted from its old row. Windows hosts are excluded. • Aggregate it with

max by (environment)

, as with the down gauge. osquery version now always reaches osctrl • Confirmation depends on the node's stored

osquery_version

, which was refreshed only from log-batch decorations. A default environment has an empty schedule with the version decorator under

always

, so that may never have updated. •

ProcessLogs

now falls back to the

version

field of the status logs when the decorations carry none. It uses the most recent valid one, validated with

ValidVersion

, so an invalid line is ignored. A decoration still wins. There is no extra database write, because the value rides the existing metadata update. API (osctrl-api) and OpenAPI • Library (super admin only):

GET /api/v1/osquery/versions

,

PUT

and

DELETE /api/v1/osquery/versions/{version}

. DELETE returns 404 for an unknown version and 409 when in use. • Environment target (environment admin):

GET

,

PUT

and

DELETE /api/v1/environments/osquery/{env}

. The GET includes host counts per version. • Node pin (admin of the node's environment):

GET

,

PUT

and

DELETE /api/v1/nodes/{env}/node/{uuid}/osquery

. The GET returns the pin, the effective target and its source, the last stored upgrade report, and

platform_unsupported

for Windows hosts. It does not compute a live resolution. • Environment admins get

available

, a names-only list of library versions, in the environment and node GETs. They never see digests. • Audit: every mutation is audited. Library audit lines list each entry as

platform/arch/format=sha256

, with

(was <old>)

when a different digest is replaced, and DELETE lists the removed entries. • Cleanup and docs: node deletion removes the pin, in both the API and the CLI.

osctrl-api.yaml

is regenerated, and only new paths and schemas are added. •

ARCHITECTURE.md

documents the route, the report field, the metrics (including the confirmation polling load against

rateLimits.osctrld

), and the

max by (environment)

guidance. Frontend • Library page (super admins only, in the nav next to Extensions): list and add versions with their entries, SHA-256 validated client-side, delete disabled while in use. • Environment "Osquery" tab: the target, a select over the available versions, a confirmation naming the version and the host count (a click alone never schedules an upgrade), clear target, and the per-version host counts. • Node "Osquery version" view: • installed version, effective target and source, and the last upgrade report with the reason in words; • the pin control behind the same lock-and-confirm pattern as flags overrides, with an optional expiry; • for no digest, the exact instruction to add it to the library. • Host-reported text is rendered as plain text only. Security and operational impact • New power: a super admin's library digest plus an environment admin's target decide what package runs as root on every host in that environment. The library is super-admin only, targets are environment-admin only and scoped to the environment in the path, pins are scoped to the node's environment, and every change is audited. • No throttle: per the design, every host upgrades at its next sync (up to an hour). The per-host pin is how an operator tests first, and each host verifies and rolls back on its own. • Untrusted input: all upgrade fields from hosts are validated before storage, and only validated values reach metric labels and audit lines. • Load: each upgrading host polls the new route about every… jmpsec/osctrl