#933 DB health canary and cached data served in `osctrl-tls` Pull request opened by javuto DB health canary + cache stale-serve for osctrl-tls Problem When

osctrl-tls

loses its database connection at runtime, the service stays alive (no fatals) but degrades to 500s on every DB-dependent endpoint — including

EnrollHandler

,

ConfigHandler

,

LogHandler

, and

QueryRead/WriteHandler

— after the ~5-minute

EnvCache

TTL expires. With a large osquery fleet, this means nodes stop getting config, can't re-enroll, and can't report logs for the duration of the outage. The service already survives the outage process-wise; it just stops being useful. Changes Add an opt-in DB health monitor to

osctrl-tls

that detects sustained DB outages and switches

EnvCache

and

SettingsCache

into stale-serve mode: cached entries are served past their TTL on a DB miss instead of returning an error, and TTLs on fresh writes are extended so the cache stays warm for the duration of the outage.

pkg/backend/health.go

(new) —

DBHealth

monitor: • Background goroutine pings the DB via

DBManager.Check()

every

Interval

(default 5s). • After

FailureThreshold

consecutive failures (default 3, ~15s),

IsDegraded()

flips to `true`; the first success clears it. •

SetOnChange(func(degraded bool))

fires on transitions — used to flip the Prometheus gauge without

pkg/backend

importing prometheus. •

nil *DBHealth

is always non-degraded, so unwired callers are unaffected.

pkg/cache/redis-json.go

+

in-memory.go

—

GetStale

method: •

RedisJSONCache.GetStale

returns a value still present in Redis regardless of TTL (Redis evicts lazily, so "stale" = past TTL but not yet evicted). •

MemoryCache.GetStale

returns a value whose TTL has expired but which the cleanup goroutine hasn't evicted yet.

pkg/environments/env-cache.go

— stale-serve + TTL extension: •

SetDBHealth(h)

wires the monitor. • On a DB miss during degradation,

GetByUUID

falls back to

GetStale

instead of erroring. • On a DB hit during degradation, writes use

envCacheDegradedTTL

(60m) instead of

envCacheTTL

(5m) so refreshed entries stay warm longer.

pkg/settings/settings-cache.go

— same stale-serve + TTL extension pattern for

RedisSettingsCache.GetMap

.

cmd/tls/main.go

— wiring: • New

--db-health-check

,

--db-health-interval

,

--db-health-threshold

flags (TLS-only, not shared with API/Admin). • When enabled, starts the monitor and wires it into

EnvCache

+

RedisSettingsCache

.

cmd/tls/handlers/metrics.go

—

osctrl_tls_db_degraded

gauge (0/1), flipped via the

OnChange

callback so external alerting can fire before the 5m TTL runs out.

deploy/config/tls.yml

— documented the three new fields with the security trade-off note.

pkg/config/types.go

+

flags.go

— added

DBHealthCheck

,

DBHealthInterval

,

DBHealthThreshold

to

YAMLConfigurationService

and registered TLS-only flags. Behavior • DB healthy (default): unchanged. Caches serve the 5m TTL as before; DB misses return errors as before. • DB degraded (≥3 consecutive ping failures):

EnvCache.GetByUUID

and

SettingsCache.GetMap

serve stale cached entries on a DB miss instead of erroring, keeping

/enroll

,

/config

,

/log

,

/distributed

responding for nodes that already have a cached env row. Fresh writes during the outage use a 60m TTL so the cache stays warm. • DB recovers: first successful ping flips the gauge back to 0 and caches resume normal TTLs. • Disabled (

--db-health-check=false

, the default): no monitor goroutine, no stale-serve, behavior identical to today. Security trade-off Serving a stale env row means a rotated enroll secret is rejected for up to the stale window. The degraded TTL is capped at 60m so rotated secrets are not accepted indefinitely. This matches the existing 5m TTL worst-case window that already exists for Redis invalidation failures, just extended for outage scenarios. Documented in

deploy/config/tls.yml

. Tests •

pkg/backend/health_test.go

— threshold/recovery/OnChange/nil-monitor/Start-Stop. •

pkg/environments/env_cache_test.go

— stale-serve on degraded, no-stale-serve when not degraded, nil health is a no-op. •

pkg/cache/in-memory_test.go

—

GetStale

returns expired-but-present value; returns false for missing keys. •

pkg/cache/redis-json_test.go

—

GetStale

returns present value; returns false for missing keys. •

pkg/settings/settings-cache_test.go

—

SetDBHealth(nil)

is a no-op. Validation •

go test ./...

— all packages pass. •

go vet ./...

— clean. •

osctrl-tls config-generate

— emits the new `dbHealthCheck`/`dbHealthInterval`/`dbHealthThreshold` fields. Not covered by tests The

RedisSettingsCache

stale-serve path is not directly tested because the in-process fake Redis store doesn't enforce TTL eviction, making it impossible to simulate "TTL expired but key still present" without a real Redis. The underlying

RedisJSONCache.GetStale

is tested directly, and the settings cache uses the same method. Risk Notes This is an opt-in, additive change behind a default-off flag. The existing 5m TTL behavior is unchanged when the flag is disabled. The stale-serve window only activates on confirmed DB degradation (3 consecutive failures), not on transient blips. Operators enabling this in production should alert on the

osctrl_tls_db_degraded

gauge so they are aware when stale data is being served. jmpsec/osctrl