Title
#fleet
Gavin

Gavin

03/23/2022, 6:34 PM
More a fleet-dev type question. Before i go down a rabbit hole I am currently looking at logging outputs for Fleet, I notice a strict 1 log message 1 write pattern for outputs however I am curious if a shared log buffer which is flushed when it reaches a certain log volume size would be beneficial to help with system throughput and scaling at the potential risk of losing log messages?
6:49 PM
Thinking about this line in the doc
Lambda is executed once per log line. As a result, queries with differential result logging might result in a higher number of Lambda invocations.
https://github.com/fleetdm/fleet/blob/0af5e161d07d46ce5e315c3a5ac4aa50ab4b0f39/docs/Using-Fleet/Osquery-logs.md I am looking at throwing together an s3 output and don’t really want to write one file per log line and would rather batch these.
7:09 PM
Actually scrap all of this I went down a rabbit hole , one should use fluentbit as a sidecar and not rewrite core software
zwass

zwass

03/23/2022, 9:33 PM
If you want to get to S3, I highly recommend Firehose. It does all the batching and scales as much as I've seen anyone need.
9:33 PM
Some forwarder in a sidecar is a totally viable option as well.
Gavin

Gavin

03/23/2022, 9:54 PM
Firehouse with a 1mb size limit can result in failed log ingestion for some badly crafted queries. Select * from installed packages for example. However I agree firehose is the most suitable for 99% of the usecases
9:56 PM
However the use of fluentbit for example which can batch multiple log lines into a single file where processing is done via a lambda function can result in a significant reduction in invocations and costs as a result
9:59 PM
I believe the actual problem I was looking to solve was how can I optimise siem throughput for cost over a separate goal of using S3 I personally got tunnel vision into sending S3 logs from fleet due to it supporting S3 as a file carving destination but the way logs are processed by the server would not lend itself to the benefit I would like to see without making foundational change s to fleets way of processing logs which in hindsight I don't believe are warranted
zwass

zwass

03/23/2022, 10:03 PM
Makes sense, thank you for sharing your reasoning!