Title
#extensions
zwass

zwass

01/26/2021, 2:46 AM
I'd be fine with writing an events buffer for osquery-go with a backing store if we could figure out how to resolve the events_optimize issue.
s

seph

01/26/2021, 2:48 AM
events_optimize
is… in many ways analogous to kafka or redis consumer groups? Now I’m giggling about creating a redis stream backed event store
zwass

zwass

01/26/2021, 2:49 AM
Maybe?
events_optimize
means each query only sees each event once (the first time the query is run after the event is generated).
s

seph

01/26/2021, 2:50 AM
Yes. If you imagined events on a log pipeline, each query is a consumer group. The underlying pipeline keeps track of that dispatch.
2:50 AM
It’s the core innovation in kafka, And something all streaming systems do now,
2:52 AM
Without having written any code… I can imagine storing an event stream, seems pretty straight forward, Borrow redis trick for naming them with timestamps. Track a pointer to the last seen item in the stream.
2:53 AM
But I kinda wish there was a pure go library for this
zwass

zwass

01/26/2021, 2:54 AM
Yeah 100% we can do that but we need a way to get the name of the executing query or some equivalent to the extension so that it can do the optimization.
2:54 AM
I've been looking at https://github.com/dgraph-io/badger as a local store for Go lately.
s

seph

01/26/2021, 3:02 AM
Do we need the name, or can you just hash the query? Slightly different semantics
3:02 AM
I bet the extension doesn’t get
LIMIT
which might break things
3:03 AM
I’ve really enjoyed github.com/dgraph-io/ristretto for a project. But it’s a bit simpler
3:05 AM
Oh, we probably don’t get the query eithger
zwass

zwass

01/26/2021, 3:05 AM
Yeah I think we don't get the query