#977 DB-backed log sinks with hot reload capabilities per environment Pull request opened by javuto DB-backed, frontend-editable log sinks with hot-reload Problem Log sinks (destinations where osctrl-tls forwards osquery status/result/query logs) were configured exclusively in the YAML

logger:

section (

deploy/config/tls.yml

). Changing a sink required editing the file and restarting

osctrl-tls

. This was the last major subsystem without a DB/frontend story, even though the surrounding plumbing (service-config table, persist-to-YAML, restart-via-service-command) already existed. Solution Replace the YAML-only sink configuration with a DB-backed, frontend-editable system that supports per-environment sinks, clone-from-environment, and hot-reload without restart. The operator UI renders a dynamic typed form per sink type — no raw JSON — and a two-step "New sink" flow (type picker → config form) keeps modals within the viewport. Architecture New package:

pkg/logsinks

•

LogSink

model (

log_sinks

table): one row per sink instance, scoped by

EnvironmentID

(0 = global fallback). Fields:

Name

,

Type

,

Enabled

,

Order

,

Config

(JSON blob),

Source

(

"service"

or

"db"

),

Info

. •

Registry

: maps each of the 11 sink types (

none

,

stdout

,

file

,

db

,

splunk

,

graylog

,

logstash

,

kinesis

,

s3

,

kafka

,

elastic

) to a

SinkSpec

containing a typed field schema (

FieldSpec

), a secret-field list, a decode function, and a build function that instantiates the existing

DataExporter

. Adding a new sink type is backend-only: implement the exporter, add the config struct, register a

SinkSpec

. •

Seed

: translates the resolved service configuration (flags, env vars, or YAML — whichever the operator used) into

LogSink

rows on first boot using create-if-missing semantics.

LoggerDBSame=true

or an empty

logger.db

block falls back to the primary DB connection. Stale seed rows from a previous boot are synced to current values; operator-edited rows (

Source="db"

) are never overwritten. • Resolution:

EffectiveFor(envID)

returns env-specific sinks when they exist, otherwise falls back to global.

BuildExportersForEnvironments

builds a

map[uint]*MultiExporter

for the TLS process. • CRUD + clone:

Create

,

Update

(with secret-merge:

"***"

placeholder preserves the existing secret),

Delete

,

CloneEnvironment

(deep-copy all sinks from one env to another, with overwrite guard). • Secret redaction:

RedactedConfig

replaces secret fields with

"***"

in read responses unless

reveal=1

is passed. Logging package:

pkg/logging

•

DataExporter

interface gained

Close() error

— called by

ReplaceExporters

on every exporter in the old set during hot-reload. Stateless sinks (Splunk, Graylog, Logstash, Elastic, Stdout, None, File) return nil; stateful sinks close their resources (Kafka closes its

*kgo.Client

, DB closes its separate connection pool when it owns one). •

LoggerTLS

is now environment-aware: holds a

map[uint]*MultiExporter

guarded by a

sync.RWMutex

.

ExportersFor(envID)

resolves env → global fallback under a read lock.

ReplaceExporters

swaps the map and closes the old set under a write lock — in-flight logs to old sinks may be dropped (warned in the UI). `Log`/`QueryLog` keep their signatures; new `LogWithEnv`/`QueryLogWithEnv` carry the numeric env ID.

ProcessLogs

and `DispatchLogs`/`DispatchQueries` thread

envID

through. Service commands:

pkg/servicecommands

• New

ActionReloadLogSinks = "reload-log-sinks"

— allowlisted and validated at request time. Consumed by `osctrl-tls`'s existing

watchServiceCommands

poller, which rebuilds the exporter map from the DB and calls

ReplaceExporters

without restarting. Service config:

pkg/serviceconfig

• Dropped the

logger

section from

SectionRegistry

(both

tls

and

api

). Log sinks are now owned by

pkg/logsinks

. Existing

service_config

rows named

logger

are left in place and ignored. API:

cmd/api/handlers/log_sinks.go

Admin-only, audit-logged, OpenAPI-annotated, gated by `service.serviceConfigEnabled`: | Method | Route | Purpose | | ------ | ---------------------------------------- | -------------------------------------------------------------------------- | | GET | /api/v1/log-sinks?env={id}&reveal={0\|1} | List (filtered by env, secrets redacted) | | GET | /api/v1/log-sinks/types | Registry: type, description, has-secret, secret fields, typed field schema | | GET | /api/v1/log-sinks/{id}?reveal={0\|1} | One sink | | POST | /api/v1/log-sinks | Create | | PUT | /api/v1/log-sinks/{id} | Update (secret merge) | | DELETE | /api/v1/log-sinks/{id} | Delete | | POST | /api/v1/log-sinks/clone | Clone env → env | | POST | /api/v1/log-sinks/apply | Queue reload-log-sinks service command | The

GET /types

response includes a

fields

array per type — a declarative schema (name, label, type, required, secret, placeholder, help, options, default) that drives the frontend's dynamic form. No per-type frontend component is needed; adding a sink type is backend-only. Frontend:

frontend/src/features/log-sinks/

• Two-step "New sink" flow: step 1 is a type-picker modal (compact grid of type buttons with per-type icons); step 2 is the config form for that type only — short for

splunk

(4 fields) or

none

(0 fields), longer for

db

(12 fields) but always bounded to one type. • Dynamic typed form (

SinkConfigFields

): renders one input per field from the schema — text for

string

, number for

integer

, checkbox for

boolean

, dropdown for

select

, password input for `password`/secret fields. Nested keys (

sasl.mechanism

) are flattened for the form and expanded back to nested JSON on submit. Secret fields are pre-filled from a reveal query when editing. • Per-type icons: a

SINK_TYPE_ICONS

map provides a distinct inline SVG pictogram for each type (trash for

none

, terminal for

stdout

, cylinder for

db

, search lens for

splunk

, bucket for

s3

, etc.), shown in the type picker, table, editor badge, and apply-confirm dialog. • "Copy from existing DB" button: in the DB sink form, fetches the

osctrl-tls

db

section from the service-config API and pre-fills all 12 fields. Case-insensitive key matching handles Go's capitalized JSON field names vs. the lowercase schema names. • Env selector: Global + each environment. Override-with-fallback semantics (env sinks replace global for that env; empty env inherits global). • Clone modal: pick source/target env, overwrite toggle. • Apply modal: warns that in-flight logs may be dropped during the hot swap; polls the service-command status until consumed. • Scrollable modals:

ModalShell

gained

bodyClassName

so the DB sink form (12 fields) scrolls within a

max-h-[70vh] overflow-y-auto

body. • Source badge: "seed" pill for service-config rows, "edited" warning pill for DB-edited rows. Bootstrap and backwards compatibility • The

logger:

section in the YAML/env/flags remains the seed source on first boot;

Seed(params, envID)

takes the fully-resolved

*config.ServiceParameters

. • Operators who never edit through the API keep running on service-config values — seed rows are synced to current values on every boot. • Once a row is edited through the API (

Source="db"

), the service-config value … jmpsec/osctrl