Alexander
05/31/2021, 6:14 PMevents_max
now closely tied with events_expiry
.
Previously events_max
meant number of events to preserve in database at cleanup. Now it's a number of batches, and batches are formed by event time.
Shouldn't this change be in 4.7.0
changelog?seph
06/01/2021, 1:21 AMtheopolis
06/01/2021, 4:02 AMevents_max
was the maximum number of events per-subscriber to keep at any point in time.
The events_expiry
controlled how often old events would be removed. Each event has a time associated and times older than the expiry window are removed.
If you do not expire events fast enough then you may hit the max and at that point the oldest events will be removed so as to not overflow the max.Alexander
06/01/2021, 2:17 PMevents_max
limits the number of unprocessed rows in table, and though osquery looses some events, it still works and doesn't trigger oom.
But with new events_max
the 50000 is too large for process events, and overflowing events eviction doesn't happen. Oom kills osquery thread, it restarts, reruns query and dies again several times untill the query gets denilisted (or continues oom-kill loop, if query is not allowed to be denilisted).
For now I decreased schedule interval on this query and events_max
to make osquery to fit into cgroup limit, but I am still searching for a better solution.event_max
should be also decreased. Here some calculations on my latest setup:
[*]osquery> SELECT
[*] ...> count(*) AS event_cnt,
[*] ...> count(DISTINCT time) AS batches_cnt,
[*] ...> DATETIME(min(time), 'unixepoch') min_time,
[*] ...> DATETIME(max(time), 'unixepoch') AS max_time
[*] ...> FROM process_events;
+-----------+-------------+---------------------+---------------------+
| event_cnt | batches_cnt | min_time | max_time |
+-----------+-------------+---------------------+---------------------+
| 75551 | 490 | 2021-05-31 21:48:22 | 2021-06-01 14:04:22 |
+-----------+-------------+---------------------+---------------------+
50000 batches is tooooo much for process_events.cleanup_events
becomes true on occasion, as it is the remaindor of sum of random values. I may be wrong, but this line is confusing me. But I am now sure if it should be triggered each 256 events or each 256 batches.theopolis
06/01/2021, 4:19 PMalessandrogario
06/01/2021, 5:04 PMAlexander
06/02/2021, 2:25 PMevents_max
limits number of batches, but I dont get how to use it, as number of events per batch is random. I could have done some statistics measures and guess new events_max
value, but here i have other problem, that gives me no guarantee that cleanup will ever happen.theopolis
06/07/2021, 2:39 AM