01/18/2023, 2:53 AM
I'm curious what this group has built from a logging pipeline out of fleet to various aggregation solutions like splunk, elastic, sumo, datadog, etc? If using splunk, are you using the new Data Manager, or the splunk add-on for aws?

Stephan M

01/18/2023, 3:57 AM
From the fleet deployment Fluent Bit sends them the logs to kafka. From there we pipeline it to any destination we want such as Splunk, Azure Data Explorer etc.

Jesus Santos

01/19/2023, 7:12 PM
Here we are sending logs from k8s pods to fluent-bit, which writes it to s3 and then we ingest it on our data lake. We will upgrade to using kafka instead of s3 for better performance soon enough tho.