I have put some comment in the

Question

< 钢铁侠> I have put some comment in the which is related to the pull request above

钢铁侠 · Answer

how about this gt I think it s why osqueryd can t expire these old logs it just can t found the data log without records log the osquery remove the `records auditeventpublisher user events` but not `data auditeventpublisher user events`

钢铁侠 · Answer

< npamnani>

npamnani · Answer

Did you try the ldb command

npamnani · Answer

sst dump file= var osquery osquery db command=scan | grep socket events | wc l will give a lot more rows than are available in the events column family

npamnani · Answer

void EventSubscriberPlugin expireCheck auto data key = data + dbNamespace auto eid key = eid + dbNamespace Min key will be the last surviving key size t threshold key = 0 auto limit = getEventsMax std vector lt std string gt keys scanDatabaseKeys kEvents keys data key if keys size lt = limit return There is an overflow of events buffered for this subscriber LOG WARNING lt lt Expiring events for subscriber lt lt getName lt lt overflowed limit lt lt limit lt lt VLOG 1 lt lt Subscriber events lt lt getName lt lt exceeded limit lt lt limit lt lt by lt lt keys size limit

npamnani · Answer

it is accessing only how many data rows are available Expiring comes later

钢铁侠 · Answer

I use `ldb db= lt path to db gt scan column family= events` it returns nothing

npamnani · Answer

try column family= queries

钢铁侠 · Answer

the size of my osquery db has arrived 1 5G and there are 753 sst files in osquery db directory I think maybe the name of the sst file should be continuous but I found the max num of the sst is 017854 sst the min is 000021 sst I think all the small num of sst file should be remove but they are not this is what I got

钢铁侠 · Answer

when I use ` column family= queries ` it has result

钢铁侠 · Answer

when I use command as follows ``` $ sst dump file= tmp osquery db 000021 sst command=scan | grep record auditeventpublisher process events $ sst dump file= tmp osquery db 000021 sst command=scan | grep data auditeventpublisher process events | wc l 4844 ```

钢铁侠 · Answer

it shows that there is no `record auditeventpublisher process events` but it has 4844 of `grep data auditeventpublisher process events` this is the `000021 sst` which is oldest sst file

钢铁侠 · Answer

< npamnani>

钢铁侠 · Answer

I think the root cause is that the osquery just delete the records auditeventpublisher socket events but not data auditeventpublisher socket events and the osquery does not remove the old sst file in time

npamnani · Answer

< 钢铁侠> scan the entire db for data auditeventpublisher socket events

npamnani · Answer

Without knowing the internals of DB we can say maybe the name of the sst file should be continuous

npamnani · Answer

what is the time stamp of this file

npamnani · Answer

last modification time

钢铁侠 · Answer

rw r r 1 user user 2138309 Jan 29 14 24 017854 sst

钢铁侠 · Answer

``` rw r r 1 user user 2136467 Jan 29 01 05 008594 sst rw r r 1 user user 2138352 Jan 29 01 05 008595 sst rw r r 1 user user 2139375 Jan 29 01 05 008596 sst rw r r 1 user user 2136993 Jan 29 01 05 008597 sst rw r r 1 user user 3278 Jan 29 01 12 008676 sst rw r r 1 user user 2140182 Jan 29 01 33 008923 sst rw r r 1 user user 2137903 Jan 29 02 05 009248 sst rw r r 1 user user 2139931 Jan 29 02 05 009249 sst rw r r 1 user user 2137200 Jan 29 02 05 009250 sst rw r r 1 user user 2139308 Jan 29 05 54 012010 sst rw r r 1 user user 2140123 Jan 29 05 54 012011 sst rw r r 1 user user 2137179 Jan 29 05 54 012012 sst rw r r 1 user user 2138895 Jan 29 05 54 012013 sst rw r r 1 user user 4779 Jan 29 05 54 012014 sst rw r r 1 user user 59646 Jan 29 07 22 012963 sst rw r r 1 user user 147760 Jan 29 08 33 013792 sst rw r r 1 user user 2136801 Jan 29 11 48 016022 sst ``` you can see that the the sst file has been removed between `02 05 009250 sst` and `05 54 012010 sst`

npamnani · Answer

what does the scan say

钢铁侠 · Answer

``` $ sst dump file= tmp osquery db command=scan | grep data auditeventpublisher socket events | wc l 42 ```

钢铁侠 · Answer

got 42 records

npamnani · Answer

not much of the data available

钢铁侠 · Answer

``` $ sst dump file= tmp osquery db command=scan | grep data auditeventpublisher process events | wc l 3433390 ```

钢铁侠 · Answer

there are a lot of process events

npamnani · Answer

did you try column family= queries

npamnani · Answer

then grep data auditeventpublisher process events

钢铁侠 · Answer

``` ldb db= tmp osquery db scan column family=queries | grep process events | wc l 51480 ```

钢铁侠 · Answer

``` ldb db= tmp osquery db scan column family=queries | grep data auditeventpublisher process events | wc l 51460 ```

npamnani · Answer

ldb db= tmp osquery db scan column family=queries | grep records auditeventpublisher process events

npamnani · Answer

What is its output

钢铁侠 · Answer

they miss 20 records is the `records auditeventpublisher process events`

钢铁侠 · Answer

the result of scaning process events is 51480 but the result of scaning data auditeventpublisher process events is 51460

钢铁侠 · Answer

the 20 missing records is all `records auditeventpublisher process events` when I compare the `process events` result and `records auditeventpublisher process events`

npamnani · Answer

did not understand

npamnani · Answer

20718897 is the last count

钢铁侠 · Answer

` ldb db= tmp osquery db scan column family=queries | grep process events | wc l` have more 20 records than ` ldb db= tmp osquery db scan column family=queries | grep data auditeventpublisher process events | wc l`

npamnani · Answer

ok got it

钢铁侠 · Answer

I want to know what the 20 records are finally I find that there are 16 records of `records auditeventpublisher process events`

钢铁侠 · Answer

and the last 4 records are ``` eid auditeventpublisher process events 0020739730 indexes auditeventpublisher process events 60 25812378 25812379 25812380 25812381 25812382 optimize process events 1548743012 optimize eid process events 0020728620 ```

npamnani · Answer

you have to count the output of ldb db= tmp osquery db scan column family=queries | grep records auditeventpublisher process events

npamnani · Answer

lt rec num gt timestamp

npamnani · Answer

compare it with the output ldb db= tmp osquery db scan column family=queries | grep data auditeventpublisher process events

钢铁侠 · Answer

``` ldb db= tmp osquery db scan column family=queries | grep data auditeventpublisher process events | wc l 51460 ``` there are 51460 <http records It|records It> is difficult to compare

npamnani · Answer

you also have to go through the logic

npamnani · Answer

db is reporting less data values than it has stored

钢铁侠 · Answer

`go through the logic` did not understand

npamnani · Answer

along with the comparison I meant

npamnani · Answer

does not matter if we are deleting the data or not

npamnani · Answer

void EventSubscriberPlugin expireCheck auto data key = data + dbNamespace auto eid key = eid + dbNamespace Min key will be the last surviving key size t threshold key = 0 auto limit = getEventsMax std vector lt std string gt keys scanDatabaseKeys kEvents keys data key if keys size lt = limit return There is an overflow of events buffered for this subscriber LOG WARNING lt lt Expiring events for subscriber lt lt getName lt lt overflowed limit lt lt limit lt lt VLOG 1 lt lt Subscriber events lt lt getName lt lt exceeded limit lt lt limit lt lt by lt lt keys size limit

npamnani · Answer

the above logic should report correct count

npamnani · Answer

but its not it mean

钢铁侠 · Answer

yep but it not right

npamnani · Answer

1 it is dangling data now

npamnani · Answer

2 it will be deleted eventually

npamnani · Answer

by the db based upon the parameter we set or its internal logic

钢铁侠 · Answer

I also read the source code ```CPP void EventSubscriberPlugin expireRecords const std string amp list type const std string amp index bool all if executedAllQueries return auto record key = records + dbNamespace auto data key = data + dbNamespace If the expirations is not removing all records rewrite the persisting std vector lt std string gt persisting records Request all records within this list size + bin offset auto expired records = getRecords list type + + index false if all amp amp expired records size gt 1 deleteDatabaseRange kEvents data key + + expired records begin gt first data key + + expired records rbegin gt first else for const auto amp record expired records if record second lt = expire time deleteDatabaseValue kEvents data key + + record first else persisting records push back record first + + std to string record second Either drop or overwrite the record list 直接设置新的rocksdb if all deleteDatabaseValue kEvents record key + + list type + + index else if persisting records size lt expired records size auto new records = boost algorithm join persisting records setDatabaseValue kEvents record key + + list type + + index new records ```

钢铁侠 · Answer

it use `auto expired records = getRecords list type + + index false ` to get the data auditeventpublisher process events I think the `getRecords ` does not return the right records or it does not find the records which they are exist indeed so the osquery can t delete the data auditeventpublisher process events

钢铁侠 · Answer

I use the ` ldb db= tmp osquery db scan column family=queries | grep records auditeventpublisher process events ` and this is what I got <https osquery slack com files UDK4HKG9Z FFR66929F image png> and this is what the ` ldb db= tmp osquery db scan column family=queries | grep data auditeventpublisher process events ` got I find that the the eid in the fisrt 10 `records auditeventpublisher process events` records are all not exist in `data auditeventpublisher process events` results Only this `records auditeventpublisher process events 60 25812380 25812383 ` and the first item in `records auditeventpublisher process events 60 25812380` is `0020688271 1548742845` I convert the `1548742845` to date which is `2019 01 29 14 20 45`

npamnani · Answer

cant delete is one thing it is not even reporting correctly how many records it has stored

钢铁侠 · Answer

``` $ ldb db= tmp osquery db scan column family=queries | grep data auditeventpublisher process events | wc l 51460 $ sst dump file= tmp osquery db command=scan | grep data auditeventpublisher process events | wc l 3433390 ```

钢铁侠 · Answer

why this two results are different

钢铁侠 · Answer

there must be something woring the size of osquery db in one of my servers has gotten to 2GB

钢铁侠 · Answer

< npamnani>