#943 S3-backed log reader for status, result, and on-demand query logs Pull request opened by javuto S3-backed log reader for status, result, and on-demand query logs Problem When the TLS logger is configured to ship logs to S3, the

osquery_status_data

,

osquery_result_data

, and

osquery_query_data

DB tables are empty — the data lives in S3 objects instead. The API handlers (

NodeLogsHandler

,

QueryResultsHandler

,

QueryResultsCSVHandler

) and the console/file-explorer managers all read from those DB tables, so the frontend could not access status logs, result logs, or on-demand query results when S3 logging was enabled. Change Introduced a

LogReader

interface with two implementations and wired the right one at API startup based on the logger config: •

dbLogReader

— the historical GORM-backed reader (legacy behavior). •

s3LogReader

— reads logs back from S3 objects written by

LoggerS3

. When the TLS logger type is

s3

, the API constructs the S3 reader from the same S3 client/bucket the TLS logger uses; otherwise it constructs the DB reader. The reader is wired into the API handlers, the console manager, and the file-explorer manager. How the S3 reader works Key layout The write path (

LoggerS3.Query

) embeds the query name in the S3 key so the reader can list by query name with a prefix filter — without it the reader would have to list every query object in the environment and decode each body to find the ones matching a name, which is catastrophically slow on a busy bucket. • status/result:

{env}/{logType}/{uuid}:{ts}.json

(unchanged) • query:

{env}/query/{name}/{uuid}:{ts}.json

(new — name is a path segment) Free chronological ordering The timestamp is the last path segment (millisecond Unix time), so lexical ordering of keys within a prefix == chronological ordering.

ListObjectsV2

returns keys lexically, so results come pre-sorted. The reader only fetches the objects for the requested page/limit — it does not download the entire prefix. • Status/result reads list by

{env}/{logType}/

prefix, keep keys whose UUID matches, apply the since/search filters, and return up to

limit

rows newest-first. • Query reads list by

{env}/query/{name}/

prefix (already name-scoped), apply the since filter, and page a contiguous slice of the sorted key list. • Streaming reads (CSV export, console/file-explorer result decoders) walk every key in the prefix and invoke the callback for each decoded row; memory is bounded by a single object body at a time. What works with S3 logging now •

GET /api/v1/logs/{type}/{env}/{uuid}

— status + result logs (paginated, searchable) •

GET /api/v1/queries/{env}/results/{name}

— on-demand query results (paginated) •

GET /api/v1/queries/{env}/results/csv/{name}

— CSV export (streaming) • Console command results + history • File explorer list/stat results + priming metadata Heatmap is unaffected The per-node activity heatmap is fed by Redis (

recordActivity

fires in the TLS ingestion handlers on every status/result/config/query_read/query_write arrival, independent of the DB logger), so it continues to populate with S3 logging. The DB-bucketed `status`/`result` categories in

computeNodeActivityForNode

are redundant with the Redis series and simply read as zero — no real gap. Files New •

pkg/logging/reader.go

—

LogReader

interface +

dbLogReader

•

pkg/logging/s3_reader.go

—

s3LogReader

(

ListObjectsV2

+

GetObject

) •

pkg/logging/s3_reader_test.go

— key/decode/ordering unit tests •

cmd/api/handlers/log_reader_test.go

— fake-reader wiring tests proving handlers route reads through the wired

LogReader

Modified •

pkg/logging/s3.go

—

LoggerS3.Query

write method with the query name in the key;

s3QueryKey

helper shared by writer and reader •

pkg/logging/logging.go

—

QueryLog

calls

l.Query

(not

l.Send

) for S3 •

pkg/console/{manager,models}.go

—

LogReader

field + `SetLogReader`;

Environment

(env UUID) stored on

Session

for the S3 key prefix;

queryResults

/

History

/

CommandResults

use the reader •

pkg/fileexplorer/{manager,models}.go

— same pattern;

RequestResults

and

RequestMetadataRows

use the reader •

cmd/api/handlers/handlers.go

—

LogReader

field +

WithLogReader

+ lazy

logReader()

fallback to

NewDBLogReader(h.DB)

•

cmd/api/handlers/logs.go

—

NodeLogsHandler

uses

h.logReader().NodeLogs

•

cmd/api/handlers/queries.go

—

QueryResultsHandler

and

QueryResultsCSVHandler

use

h.logReader().QueryResults

/

StreamQueryResults

•

cmd/api/main.go

— construct S3 or DB reader based on `flagParams.Logger.Type`; wire into handlers + console + file-explorer managers Backward compatibility • When no

LogReader

is wired, handlers and managers fall back to

NewDBLogReader(h.DB)

, so existing tests that only set

WithDB

keep working unchanged. • The

Environment

column added to

console_sessions

and

file_explorer_sessions

is additive and migrated via the existing

AutoMigrate

in each

NewManager

. • The S3 query-log key layout change applies to new writes only; existing objects written by the old

Send

path are not listed by the new query reader (they predate this change and would not have been readable by name regardless). Validation •

go build ./...

— clean •

go test ./...

— all pass, including new S3 reader and LogReader-wiring tests •

golangci-lint

on touched packages — only pre-existing issues remain •

npm run check

(tsc) — clean •

npm run test

(vitest) — 190/190 frontend tests pass jmpsec/osctrl