:wave: folks. I work on Kolide @ 1Password. I was ...
# core
b
šŸ‘‹ folks. I work on Kolide @ 1Password. I was digging into a few devices with hundreds of gigabytes of rocksdb SST files without anything in osquery to justify them. I think I narrowed it down to the events (
queries
) column family compaction? These devices have high volumes of process events (1k-8k/s). I reproed locally, and the SST files remaining were cleaned up once I compacted by hand. github.com/osquery/osquery/commit/e0387374c55c… would make sense to me, but I don't know why we skipped kEvents šŸ‘€? 6606 doesn't say either.
z
How long does the startup take with this compaction? Seems to make sense to me but that's my biggest potential concern.
b
I was working on a build with a database to answer that. Let me get back with some timings šŸ¤”.
I had wondered if performance was the concern.
On a decent Linux machine, osquery with a 2GB database takes ~40ms from run to socket. The database compacts to 149MB, taking ~9.5s to socket. Subsequent starts took ~300ms before a socket. I also tried some larger synthetic databases with tens of millions of live events, compaction costs stay high even when little is compacted.
So this feels backwards šŸ¤”. I'm going to dig into rocksdb compaction settings. They've got a bunch of knobs.
z
That makes sense to me... Probably has to scan the entire DB to do compaction.
šŸ‘ 1
b
github.com/osquery/osquery/pull/9251 I dumped most of what I learned into the summary. I think configuring a TTL is pretty pragmatic. Thanks for the nudge about startup time impact.