#934 Resist backend outage via DB health canary in `osctrl-api` Pull request opened by javuto osctrl-api: resist backend outage via DB health canary + EnvCache stale-serve Problem

osctrl-api

had no defense against transient backend outages: every request that resolved an environment hit the DB through

EnvCache

, and a DB miss returned 500. A short DB blip would take down read-only API routes (env lookups, platforms, audit, etc.) for the whole fleet.

osctrl-tls

already solved this in

resistant-backend-outage-tls

(commit

8dafca90

) by adding a

pkg/backend.DBHealth

canary that pings the DB on a fixed interval and, after N consecutive failures, flips `EnvCache`/`SettingsCache` into stale-serve mode (serve cached entries on DB miss, extend TTLs to ~60m). Change Ports the same pattern to

osctrl-api

, reusing the now-shared

pkg/backend.DBHealth

and the `EnvCache.SetDBHealth`/`GetStale` plumbing already merged in the TLS branch. •

pkg/config/flags.go

— Adds

--db-health-check

,

--db-health-interval

,

--db-health-threshold

(and matching

DB_HEALTH_*

env vars) to

InitAPIFlags

. The

YAMLConfigurationService

struct already had the

DBHealth*

fields from the TLS branch, so

types.go

needed no change. •

cmd/api/main.go

— After

NewRedisEnvCache

, conditionally constructs

backend.NewDBHealth(db, interval, threshold)

, starts the monitor, and wires it via

envCache.SetDBHealth(dbHealth)

. On a DB outage,

EnvCache.GetByUUID

now serves the stale cached env row instead of 500, and writes during degradation use

envCacheDegradedTTL

(60m) so the cache stays warm. •

deploy/config/api.yml

— Documents the three new

dbHealth*

keys under

service:

with the same semantics as

tls.yml

. Scope and limitations • Read-only paths only. Env CRUD, query runs, carve runs, and other write paths still hit the DB directly and will fail during an outage — by design. The canary protects the env-resolution read path that the API's read handlers depend on. • No settings cache.

osctrl-tls

also wires `RedisSettingsCache.SetDBHealth`; osctrl-api does not currently use

RedisSettingsCache

, so there is no settings-cache half to wire here. Adding one would be a separate change. • No Prometheus gauge. osctrl-api has no metrics surface, so the

SetDBDegraded

callback from the TLS branch is not wired.

IsDegraded()

is still consumed by EnvCache; if a metrics endpoint is added to the API later, the same callback pattern applies. • Bounded stale window. The degraded TTL is capped at 60m so rotated enroll secrets are not accepted indefinitely — the same trade-off documented in the TLS branch. Validation •

go build ./cmd/api/...

— clean •

go vet ./cmd/api/...

— clean •

go test ./pkg/backend ./pkg/environments ./pkg/cache ./pkg/config ./pkg/settings

— all pass (covers the

DBHealth

canary,

EnvCache

stale-serve, and

GetStale

cache fallbacks added in the TLS branch) No new tests were added; the behavioral change in the API is a 7-line wiring block in

cmd/api/main.go

exercising code paths already covered by the TLS branch's tests in `pkg/backend`/`pkg/environments`. jmpsec/osctrl