Title
#extensions
s

Seshu

01/26/2021, 3:46 AM
map<string, Object>
where
Object
is a
union
should work in thrift AFAIK. Overhead for serialization/de-serialization via domain sockets should be negligible compared to JSON. We have been pushing events in near real-time for ~4 years. OSS PR https://github.com/osquery/osquery/pull/3482 and others didn't go anywhere. Our Osquery at most have a latency of 4 seconds on events. Here is a customer from 2020 @scale conference who measured events -> alerts in ~1.7 seconds

https://www.youtube.com/watch?v=-9BfXMYn0wk&amp;t=1210

The more I explore extensions, these limitations are biting me. Happy to contribute, if there is consensus.
zwass

zwass

01/26/2021, 4:25 PM
Regarding events -- did you ever do an analysis of the perf cost of the rocksdb roundtrip? I'd be curious to see what that looked like.
s

Seshu

01/26/2021, 4:37 PM
We probably did some perf testing in lab few years back. Will see if I can find the details. We haven't been writing event to DB for few years now. Helps with perf and resource utilization. We use it as fallback when remote TLS is not reachable.
theopolis

theopolis

01/26/2021, 6:18 PM
I think there’s a good case for sending events directly to the logger if that’s what you are advocating for. I think we need to win over Seph’s strong opinions but my feedback/requirement is that the original PR is hacky and we need to spend more time on the interface.
s

seph

01/27/2021, 10:07 PM
ToB and Teddy have both done rework on rocksDB. Alessandro says it’s a lot faster now.
10:09 PM
My general sentiment is that I’m not sure if this belongs in osquery. At least today, I feel less strident than some days, but I think osquery fufills a different niche than the Elastic Beats tools. And what you generally describe feels like trying to push beats into osquery, and I don’t really understand why.
10:10 PM
So even before the interfaces, I feel like we should settle on the why of it.
2:45 AM
Thinking a bit — I could imagine a streaming sql (<cough> CEP) version of osquery. But that feels like a big project.