#947 Improvements in fake_news_go to be able to test 10K clients Pull request opened by javuto Scale fake_news_go to 10,000 osquery clients Problem

fake_news_go

could run at 10,000 nodes, but the measurements were polluted by client-side bottlenecks rather than reflecting server-side capacity: • HTTP connection pool starvation (default

MaxIdleConnsPerHost

of 2) • Unbounded

queryWrite

goroutine spawn on query floods (thundering herd against

/write

) • Sequential enrollment (10k sequential HTTP round-trips to start) • Non-atomic state file writes (crash corrupts state, loses all node keys) • Nested mutex on every request (contention bottleneck at scale) • Unbounded response body reads (memory exhaustion risk) Changes High-impact (correctness at 10k scale) 1. HTTP transport tuned for high concurrency (

internal/transport/client.go

) •

MaxIdleConns: 2000

,

MaxIdleConnsPerHost: 2000

— the default of 2 per host was orders of magnitude too small; every request was either waiting for a connection or opening a new one with a TLS handshake, so measured latency reflected client-side connection starvation rather than server behavior. •

DialContext

with 10s dial timeout + 30s keepalive. •

ForceAttemptHTTP2: true

for connection reuse. • Response body capped at 4 MiB via

io.LimitReader

to prevent a malformed or hostile server response from exhausting memory across 10k concurrent nodes. 2. Per-node queryWrite semaphore (

fake_news.go

) •

queryRead

now spawns

queryWrite

goroutines through a buffered channel of size 2 per node. Previously a query flood to 10k nodes spawned 10k+ transient goroutines hitting

/write

simultaneously — a thundering herd that doesn't reflect real osquery behavior, which processes queries serially per node. If the cap is reached, the query is skipped for that cycle (non-blocking, no deadlock risk). 3. Concurrent enrollment with bounded parallelism (

fake_news.go

) •

enrollNodes

now enrolls with up to 100 concurrent goroutines instead of a sequential loop. At 10k nodes with 50ms per enroll, startup drops from ~8 minutes to ~5 seconds. Medium-impact (reliability + contention) 4. Atomic state file writes (

fake_news.go

) •

saveNodesToFile

now writes to a temp file in the same directory and renames on success. A crash or signal during the 10-second save tick no longer leaves a truncated/corrupt JSON file. Previously this would force all 10k nodes to re-enroll with new UUIDs on restart, losing all accumulated node keys. 5. Reduced lock contention (

fake_news.go

) •

RecordURLOperation

no longer takes the outer

<http://GlobalStats.mu|GlobalStats.mu>

mutex — only the per-URL mutex. The nested double lock on every request was a contention bottleneck at 10k nodes.

lastUpdate

is updated under the URL mutex. Verified not an issue Sweep stage overlap — re-read the flow in `runSweep`:

cancel()

at the end of each stage iteration stops the previous stage's goroutines before the next stage starts. No overlap exists; no fix needed. Files changed •

tools/fake_news_go/internal/transport/client.go

— tuned transport, response body cap •

tools/fake_news_go/fake_news.go

— queryWrite semaphore, concurrent enrollment, atomic state writes, reduced locking Validation •

go build ./tools/fake_news_go/...

— clean •

go test ./tools/fake_news_go/...

— all pass (8 packages) •

go test ./...

— full suite passes •

gofmt

applied to changed files jmpsec/osctrl