I d be fine with writing an events buffer for osquery go wit osquery #extensions

I'd be fine with writing an events buffer for osqu...

zwass

01/26/2021, 2:46 AM

I'd be fine with writing an events buffer for osquery-go with a backing store if we could figure out how to resolve the events_optimize issue.

💯 1

seph

01/26/2021, 2:48 AM

events_optimize

is… in many ways analogous to kafka or redis consumer groups? Now I’m giggling about creating a redis stream backed event store

zwass

01/26/2021, 2:49 AM

Maybe?

events_optimize

means each query only sees each event once (the first time the query is run after the event is generated).

seph

01/26/2021, 2:50 AM

Yes. If you imagined events on a log pipeline, each query is a consumer group. The underlying pipeline keeps track of that dispatch.

seph

01/26/2021, 2:50 AM

It’s the core innovation in kafka, And something all streaming systems do now,

👍 1

seph

01/26/2021, 2:52 AM

Without having written any code… I can imagine storing an event stream, seems pretty straight forward, Borrow redis trick for naming them with timestamps. Track a pointer to the last seen item in the stream.

seph

01/26/2021, 2:53 AM

But I kinda wish there was a pure go library for this

zwass

01/26/2021, 2:54 AM

Yeah 100% we can do that but we need a way to get the name of the executing query or some equivalent to the extension so that it can do the optimization.

zwass

01/26/2021, 2:54 AM

I've been looking at https://github.com/dgraph-io/badger as a local store for Go lately.

seph

01/26/2021, 3:02 AM

Do we need the name, or can you just hash the query? Slightly different semantics

seph

01/26/2021, 3:02 AM

I bet the extension doesn’t get

LIMIT

which might break things

seph

01/26/2021, 3:03 AM

I’ve really enjoyed github.com/dgraph-io/ristretto for a project. But it’s a bit simpler

seph

01/26/2021, 3:05 AM

Oh, we probably don’t get the query eithger

zwass

01/26/2021, 3:05 AM

Yeah I think we don't get the query

Open in Slack

Previous Next