#988 Replace the failed-enrolls counter with reported osquery errors Pull request opened by javuto Replace the failed-enrolls counter with reported osquery errors The dashboard's "Failed enrolls (24h)" tile read the audit log and filtered client-side for lines starting with

failed enroll

. It's replaced by "Reported errors (24h)" — status logs the fleet sent at osquery's ERROR severity — surfaced across the dashboard, the nodes table and the node detail page. Errors only. osquery's ladder is 0=INFO, 1=WARNING, 2=ERROR, and warnings are routine enough (a table unavailable on the platform, a transient permission issue) that counting them would keep the tile permanently lit and train people to ignore it. Counted at ingest, not at read time The obvious implementation —

COUNT(*) FROM osquery_status_data WHERE created_at > now()-24h AND severity != '0'

— breaks on three things: 1. The logs may not be in the database. With

logger.type

set to

s3

,

splunk

,

kafka

,

graylog

or

elastic

, that table is empty and the tile would read a permanent zero. 2. No index supports it.

OsqueryStatusData

indexes only `UUID`;

gorm.Model

doesn't index

CreatedAt

. That's a full scan of the highest-volume table, every dashboard load, every 5 minutes. 3.

Severity

is stored as a string, so

severity > '1'

is a string comparison — an easy silent bug. Instead osctrl-tls tallies ERROR entries per status batch and increments the existing Redis activity rollup (

pkg/activity

). O(1) per log, correct under every log sink, and the dashboard read becomes O(1). The count runs inside the goroutine that already calls

ProcessLogs

, so it stays off the request hot path. No key migration.

bitOffset

is linear in the event-type index and

decodeDay

is bounds-checked, so appending

EventStatusError

at index 6 leaves every existing counter at its offset — old blobs simply decode the new type as zero.

EventTypeCount

now carries a comment recording why new types must be appended, never inserted. Drill-down: which nodes are erroring Reading every node's series to answer this would scale with fleet size, for a question whose answer is a handful of nodes. The write path also does a

ZINCRBY

into a per-env, per-day sorted set, so

TopErrorNodes

is a

ZREVRANGE

— cost scales with the number of erroring nodes. Only errors pay for it, and only when there are any.

GET /api/v1/stats/activity/error-nodes/{env}

returns the worst 10 with hostnames resolved; a node deleted since it errored keeps its row and shows by UUID. Clicking the tile opens a dialog listing them, each linking to the node. The tile only becomes a

<button>

when there's something behind it — keyboard-reachable and announced as interactive — and stays a plain

<div>

at zero. Where errors show up | Surface | What was added | | -------------------------------- | ------------------------------------------------------------------------------------ | | Dashboard tile | "Reported errors (24h)", clickable into the drill-down | | Dashboard chart | A fifth line in bright red, in the existing recolourable palette | | Endpoint health panel | An "Errors" row, red row wash when non-zero | | All nodes table | A fifth heatmap lane, red, with its own intensity scale | | Node detail heatmap | An errors row alongside status/result/query/config, plus a count badge in the header | | Node detail "Endpoint last seen" | The Errors row goes red and reads "N errors in 24h" | New

--error-bright

token (

#ff4d4f

dark /

#f01c1c

light), deliberately hotter than

--danger

, which is already used for softer degraded-but-expected states. Every red surface stays neutral at zero — a permanently red panel is one nobody reads. Errors are excluded from every "total" (activity

Total

series, the heatmap's

totalEvents

): they're a subset of the status traffic already counted, so including them would claim the node sent more events than it did. Bugs caught during the work •

ReadSeries

had a duplicated fill loop. The per-node path didn't share

fillSeries

with

ReadEnvSeries

, so per-node error counts would have silently read zero — and the drill-down is built on exactly that path. Both now share it. • The heatmap merge was gated on

config

.

mergeNodeActivityBuckets

bounded its alignment loop by `config.length`; it now takes the max across all Redis series, so a payload with

status_error

but no

config

still aligns instead of reading zero. Changes •

pkg/activity

—

EventStatusError

,

StatusError

series,

ErrorRankKey

,

TopErrorNodes

•

cmd/tls/handlers

—

countStatusErrors

,

recordActivityCount

, wired into the log POST path •

cmd/api/handlers/stats.go

,

cmd/api/main.go

—

EnvErrorNodesHandler

+ route •

frontend/

— API client, dashboard tile/chart/panel, nodes-table lane, node-detail row,

--error-bright

•

osctrl-api.yaml

regenerated Testing • Go: severity filtering (int and quoted-string forms — osquery sends both), malformed-batch tolerance (returns 0 rather than failing ingestion), batch counts advancing by N, zero counts emitting no event, ranking across the UTC day boundary a 24h window straddles, env isolation, non-error events staying out of the ranking, limit truncation, and

Total

not double-counting. • Frontend: tile sum and label, inert-at-zero, drill-down dialog, chart line present with no markers, heatmap lane cell count (96 → 120, commented with the lane math), merge alignment including the omitted-field case. • The fake Redis in

pkg/activity

tests gained `ZINCRBY`/`ZREVRANGE` support — it's a hand-rolled RESP server that only implemented the commands used so far. • Full suite green: `go build`/`go vet`/`go test`,

make openapi-check

, frontend 265 tests,

tsc

. jmpsec/osctrl