https://github.com/osquery/osquery logo
Title
g

Gavin

03/23/2022, 6:34 PM
More a fleet-dev type question. Before i go down a rabbit hole I am currently looking at logging outputs for Fleet, I notice a strict 1 log message 1 write pattern for outputs however I am curious if a shared log buffer which is flushed when it reaches a certain log volume size would be beneficial to help with system throughput and scaling at the potential risk of losing log messages?
Thinking about this line in the doc
Lambda is executed once per log line. As a result, queries with differential result logging might result in a higher number of Lambda invocations.
https://github.com/fleetdm/fleet/blob/0af5e161d07d46ce5e315c3a5ac4aa50ab4b0f39/docs/Using-Fleet/Osquery-logs.md I am looking at throwing together an s3 output and don’t really want to write one file per log line and would rather batch these.
Actually scrap all of this I went down a rabbit hole , one should use fluentbit as a sidecar and not rewrite core software
👀 1
z

zwass

03/23/2022, 9:33 PM
If you want to get to S3, I highly recommend Firehose. It does all the batching and scales as much as I've seen anyone need.
Some forwarder in a sidecar is a totally viable option as well.
g

Gavin

03/23/2022, 9:54 PM
Firehouse with a 1mb size limit can result in failed log ingestion for some badly crafted queries. Select * from installed packages for example. However I agree firehose is the most suitable for 99% of the usecases
However the use of fluentbit for example which can batch multiple log lines into a single file where processing is done via a lambda function can result in a significant reduction in invocations and costs as a result
I believe the actual problem I was looking to solve was how can I optimise siem throughput for cost over a separate goal of using S3 I personally got tunnel vision into sending S3 logs from fleet due to it supporting S3 as a file carving destination but the way logs are processed by the server would not lend itself to the benefit I would like to see without making foundational change s to fleets way of processing logs which in hindsight I don't believe are warranted
z

zwass

03/23/2022, 10:03 PM
Makes sense, thank you for sharing your reasoning!