Still inverstigating changes in event framework from 4 7 0 a

Question

Still inverstigating changes in event framework from 4 7 0 and 4 8 0 Seems like `events max` now closely tied with `events expiry` Previously `events max` meant number of events to preserve in database at cleanup Now it s a number of batches and batches are formed by event time Shouldn t this change be in `4 7 0` changelog

seph · Answer

All changes should be in the CHANGELOG yes But they may not be written to clearly convey complex effects This is somewhat because it s a complex system And somewhat because I write the CHANGELOG and don t always get it right And somewhat because the PRs themselves are not always very clear

theopolis · Answer

The two were always closely related The intention of `events max` was the maximum number of events per subscriber to keep at any point in time The `events expiry` controlled how often old events would be removed Each event has a time associated and times older than the expiry window are removed If you do not expire events fast enough then you may hit the max and at that point the oldest events will be removed so as to not overflow the max

theopolis · Answer

4 8 0 fixed a regression introduced between 4 6 0 and 4 7 0 meaning the regression showed up as a bug in 4 7 0

Alexander · Answer

I agree with that but this change have broken some assumptions that we used for out installation so I wanted to discuss it and warn other osquery users I wanted to warn that there are backward incompatible change in osquery that ve been brought with eventing framework refactor and leads to some unexepected side effects if not handled properly As an example I can describe a piece of out infrastructure We use osquery to monitor linux hosts To ensure that osquery won t pass allowed limits on memory and cpu usage we are using softlimit by osquery watchdog and hardlimit by cgroup Rarely we have burst proccess events activity on some of that hosts that can make osquery scheduler to pass the cgroup limit on processing query and to be killed Older `events max` limits the number of unprocessed rows in table and though osquery looses some events it still works and doesn t trigger oom But with new `events max` the 50000 is too large for process events and overflowing events eviction doesn t happen Oom kills osquery thread it restarts reruns query and dies again several times untill the query gets denilisted or continues oom kill loop if query is not allowed to be denilisted For now I decreased schedule interval on this query and `events max` to make osquery to fit into cgroup limit but I am still searching for a better solution

Alexander · Answer

BTW I think default `event max` should be also decreased Here some calculations on my latest setup ``` osquery gt SELECT gt count AS event cnt gt count DISTINCT time AS batches cnt gt DATETIME min time unixepoch min time gt DATETIME max time unixepoch AS max time gt FROM process events + + + + + | event cnt | batches cnt | min time | max time | + + + + + | 75551 | 490 | 2021 05 31 21 48 22 | 2021 06 01 14 04 22 | + + + + +``` 50000 batches is tooooo much for process events

Alexander · Answer

< theopolis> if haven t please reads this also It looks like that `cleanup events` becomes true on occasion as it is the remaindor of sum of random values I may be wrong but this line is confusing me But I am now sure if it should be triggered each 256 events or each 256 batches

Alexander · Answer

s now not slightly smiling face Sorry for some errors and misstypes in my messages

theopolis · Answer

Ping < alessandrogario>

theopolis · Answer

Sorry meant to ping Alessandro for clarification about the batching

alessandrogario · Answer

I believe the old implementation was splitting batches in half while the new implementation evicts entire batches

alessandrogario · Answer

that was causing all the indexes to be split rewritten in the database

alessandrogario · Answer

Alexander · Answer

Sorry < alessandrogario> but I didn t got what you wanted to say Old implementation pre 4 7 0 limited the number of events in database and limited the number of rows processed by query at one run That could be used to do some assumptions and limit resource usage of osquery by dropping overflowing events New `events max` limits number of batches but I dont get how to use it as number of events per batch is random I could have done some statistics measures and guess new `events max` value but here i have other problem that gives me no guarantee that cleanup will ever happen

theopolis · Answer

I agree with < Alexander> I put up the change here