#979 Per-sink data tracking with zero-DB-impact hot path Pull request opened by javuto Per-sink data tracking with zero-DB-impact hot path Problem There was no visibility into how much data each log sink was actually receiving. Operators could configure sinks but had no way to tell which ones were active, how many logs they were processing, or how much data was flowing to each destination. Solution Add per-sink byte and export-count tracking that uses lock-free atomic counters on the hot path (every

Export

call) and a background writer that flushes to the database every 30 seconds — the same batch-writer pattern already used for node

last_seen

updates. The DB is never touched during log ingestion; only

atomic.Int64

adds happen, and the periodic flush is at most N sinks (typically 1-5) every 30 seconds. Architecture Hot path:

CountedExporter

wrapper (

pkg/logging/exporter.go

) • New

SinkStats

struct holds per-sink

atomic.Int64

counters (

BytesSent

,

ExportCount

) linked to the sink's DB row ID. •

CountedExporter

wraps any

DataExporter

and increments the counters before delegating to the inner exporter. The wrapping is transparent — the underlying exporter receives the call unchanged. •

MultiExporter

gained a

stats []*SinkStats

slice and a

NewMultiExporterWithStats

constructor that wraps each exporter in a

CountedExporter

.

Stats()

returns the slice for snapshotting. •

BuildExporters

in

pkg/logsinks

now uses

NewMultiExporterWithStats

, passing each sink's DB row ID so the counters are linked back to the

log_sinks

row. Background persistence:

SinkStatsWriter

(

pkg/logsinks/logsinks.go

) • A background goroutine (started in

cmd/tls/main.go

, 30s interval) snapshots all

SinkStats

from every live

MultiExporter

via

LoggerTLS.AllExporters()

— a read-locked shallow copy of the exporter map — and issues one

UPDATE log_sinks SET bytes_sent=..., exports_count=...

per sink. • On shutdown, a final flush ensures the last window of stats is not lost. • The writer automatically picks up new sinks after a hot reload because it reads the current exporter map each tick.

LoggerTLS.AllExporters()

(

pkg/logging/logging.go

) • New method returns a shallow copy of the entire

map[uint]*MultiExporter

under a read lock, so the stats writer can iterate without holding the lock during the DB flush. DB columns (

pkg/logsinks/logsinks.go

) •

LogSink

gained

BytesSent int64

and

ExportsCount int64

(both

gorm:"default:0"

), auto-migrated. API (

cmd/api/handlers/log_sinks.go

) •

logSinkDTO

gained

bytes_sent

and

exports_count

fields, populated from the DB row in

toLogSinkDTO

. All list/get endpoints return the stats. Frontend (

frontend/src/features/log-sinks/LogSinksPage.tsx

) • New "Data sent" column in the sink table (right-aligned,

font-mono-tabular

), using the existing

formatBytes()

helper to show human-readable sizes (B / KB / MB / GB). Shows

—

when no data has been sent. Tooltip shows the raw export count. •

LogSink

TS interface gained

bytes_sent

and

exports_count

. Validation • Go: 44 packages pass, 0 failures. New tests: • `TestBuildExportersTracksStatsPerSink`: verifies that two

Export

calls increment both sinks' atomic counters to the expected bytes/count, and that

UpdateSinkStats

persists them to the DB. • `TestDisabledSinkNotCounted`: verifies disabled sinks are excluded from stats tracking. • Frontend: 240 tests pass, type check clean. • Existing

TestBuildExportersSkipsDisabledAndUnknown

and

TestBuildExportersForEnvironmentsGroupsByEnv

updated for the new

NewMultiExporterWithStats

constructor (transparent — same

ExporterNames()

behavior). Files Modified (8 files): •

pkg/logging/exporter.go

—

SinkStats

,

CountedExporter

,

NewMultiExporterWithStats

,

ExporterEntry

,

Stats()

•

pkg/logging/logging.go

—

AllExporters()

method •

pkg/logsinks/logsinks.go

— `BytesSent`/`ExportsCount` columns,

SinkStatsWriter

,

UpdateSinkStats

,

BuildExporters

uses

NewMultiExporterWithStats

•

pkg/logsinks/logsinks_test.go

— stats tracking + disabled-sink tests •

cmd/tls/main.go

— start/stop

SinkStatsWriter

•

cmd/api/handlers/log_sinks.go

— `bytes_sent`/`exports_count` in DTO •

frontend/src/api/log-sinks.ts

— `bytes_sent`/`exports_count` in

LogSink

interface •

frontend/src/features/log-sinks/LogSinksPage.tsx

— "Data sent" column with

formatBytes

jmpsec/osctrl