https://github.com/osquery/osquery logo
Title
i

Ivanlei

01/18/2020, 3:02 PM
• we use benchmarks to create artificial constant rate of process events per second • we monitor CPU, RAM, & diskIO • we monitor the osquery logs to see when event buffers overflow one finding i wanted to share - we set
snapshot=true
on our eventing queries and measured a 5X increase in the events per second throughput of
process_events
before we saw events buffers overflow
t

theopolis

01/18/2020, 3:06 PM
Do your queries on
process_events
look similar to
select * from process_events
or are they more complex?
i

Ivanlei

01/18/2020, 3:06 PM
they are more complex. we join on stuff like process table & process_containers table
those joins seemingly have no real perf impact
t

theopolis

01/18/2020, 3:07 PM
ah, cool, I asked because I think there's some under the hood logic to detect simple queries against
_events
tables and turn on
snapshot=True
for you (I am going from memory though)
i

Ivanlei

01/18/2020, 3:08 PM
we have about the same performance whether we join to the tables or not
🤔 1
t

theopolis

01/18/2020, 3:08 PM
this is worth investigating, thanks for sharing the analysis!
z

zwass

01/18/2020, 6:02 PM
One thing that this brings up for me... If someone has
events_optimize
on, then we can probably skip the diffing step on a differential query to an event-based table... the results should all be "added".
i

Ivanlei

01/18/2020, 6:03 PM
@zwass - kinda naively assumed that was how things were working. we had
removed=true
on our event queries and had
events_optimize=true
. it surprised me how much of a difference
snapshot=true
made for the audit data.
z

zwass

01/18/2020, 6:06 PM
The major difference in perf I would anticipate for snapshot vs. differential would be the diffing step. Both generate the results in approximately the same way. Someone would have to have special-cased the diffing for event-based tables -- I'm not sure that has been done.
i

Ivanlei

01/18/2020, 6:08 PM
my read says it hasn’t been
z

zwass

01/18/2020, 6:09 PM
Seems ripe for doing!
i

Ivanlei

01/18/2020, 6:16 PM
there is code related to
events_optimize
that introspects the query on a specific event table. potentially a cheap spot to simply flip the
snapshot
option on for the
ScheduledQuery
sounds cleaner in some respects than special-casing the diffing logic
z

zwass

01/18/2020, 6:26 PM
It's an interesting idea. snapshot vs. diff queries have different log schema so that could be an issue.
i

Ivanlei

01/18/2020, 8:58 PM
a doc update recommending
snapshot
would even go a long way
people who care about event perf have read every doc