I'm curious what this group has built from a logging pipeline out of fleet to various aggregation solutions like splunk, elastic, sumo, datadog, etc? If using splunk, are you using the new Data Manager, or the splunk add-on for aws?
01/18/2023, 3:57 AM
From the fleet deployment Fluent Bit sends them the logs to kafka. From there we pipeline it to any destination we want such as Splunk, Azure Data Explorer etc.
01/19/2023, 7:12 PM
Here we are sending logs from k8s pods to fluent-bit, which writes it to s3 and then we ingest it on our data lake.
We will upgrade to using kafka instead of s3 for better performance soon enough tho.