Title
#general
v

Vikram

08/06/2020, 12:53 PM
How do we force osquery to expire old data in case of no connection between the osquery endpoint and the host server. For example if there is no connection between the server and the endpoint for say 12 hours, how do we make sure osquery will return the latest ntfs_journal_events data when the connection is re-established? Right now when doing FIM, if connection is lost for some time and then returns back osquery which start with the oldest data and will lag for multiple hours. I have tried using --events_expiry but did not get the desired results.
theopolis

theopolis

08/06/2020, 2:36 PM
Hey @Vikram, let me know if I understand the situation. osquery is running and collecting events, for example from ntfs_journal_events, for a period of 12 hours. Specifically, the machine is on, and being used, and generating events - but the machine does not have internet access. Thus a large amount of events are cached for logging. When internet access is restored osquery will start logging all of the cached events. But since there are many, and since logging happens in batches, it might take a while to log everything that cached over that 12hour period.
2:38 PM
Similarly, let me know if I understand the problem. When a machine comes online after a long period you want to know the most recent events first. If that machine has to work in batches to log a lot of cached data then it might take hours to catch up.
v

Vikram

08/06/2020, 2:40 PM
You understand the problem nicely, so how do I make it so we get the latest data or forcibly expire the old data so it does not take hours to catch up
zwass

zwass

08/06/2020, 3:31 PM
--buffered_log_max
v

Vikram

08/06/2020, 3:39 PM
thank you I will try this and get back to you
1:31 PM
Works great. Thank you! Sorry for the late response