`map lt string Object gt ` where `Object` is a `union` shoul

Question

`map lt string Object gt ` where `Object` is a `union` should work in thrift AFAIK Overhead for serialization de serialization via domain sockets should be negligible compared to JSON We have been pushing events in near real time for ~4 years OSS PR and others didn t go anywhere Our Osquery at most have a latency of 4 seconds on events Here is a customer from 2020 scale conference who measured events gt alerts in ~1 7 seconds The more I explore extensions these limitations are biting me Happy to contribute if there is consensus

zwass · Answer

Regarding events did you ever do an analysis of the perf cost of the rocksdb roundtrip I d be curious to see what that looked like

Seshu · Answer

We probably did some perf testing in lab few years back Will see if I can find the details We haven t been writing event to DB for few years now Helps with perf and resource utilization We use it as fallback when remote TLS is not reachable

theopolis · Answer

I think there s a good case for sending events directly to the logger if that s what you are advocating for I think we need to win over Seph s strong opinions but my feedback requirement is that the original PR is hacky and we need to spend more time on the interface

seph · Answer

ToB and Teddy have both done rework on rocksDB Alessandro says it s a lot faster now

seph · Answer

My general sentiment is that I m not sure if this belongs in osquery At least today I feel less strident than some days but I think osquery fufills a different niche than the Elastic Beats tools And what you generally describe feels like trying to push beats into osquery and I don t really understand why

seph · Answer

So even before the interfaces I feel like we should settle on the why of it

seph · Answer

Thinking a bit I could imagine a streaming sql lt cough gt CEP version of osquery But that feels like a big project