Is there any benefit from having two of the same queries run

Question

Is there any benefit from having two of the same queries running at different intervals one that is set to `snapshot` and the other set to `differential` Or would it make more sense to just have `differential` on a more frequent schedule

zwass · Answer

Some folks see benefit from this Snapshot queries are easy to interpret since they show the full state at once Of course they generate more data to store in wherever your logging pipeline goes Some folks do things like schedule a snapshot query for once a day and the same query as differential every 10 minutes intervals just for example and then you can reconstruct the state at any given time by applying the diffs to the snapshot

seph · Answer

Im not sure that works in practice The diffs aren t based on the scheduled results so weird skew is possible

seph · Answer

I suspect people use snapshot to try to capture state and diffs to alert in specific conditions But shrug I m speculating

Alexander · Answer

We are using such scheme in our installation In our model data loss is possible and happens sometimes in logs collector so daily `snapshot` covers possible loss off `differential` query result for some queries where we want to be able to rebuild state of machine for example deb packages query

Alexander · Answer

But we don t use distributed queries If you have and can use them and don t need timelines then you can consider to use fleet or kolide service to get the most current state of machine

seph · Answer

How do you reconcile the diffs with the snapshots Or is your data seldom changing so it s a non issue

seph · Answer

How do you re sync a diff stream that has dropped messages

Alexander · Answer

Yes frequent changes require other pipeline and events tables The scheme I described is for seldom changed like installed packages kernel modules users list and it is applied to servers Usually it is enough to just take latest from added and snapshot result in some time window But if you want to use such pipeline in automation and smart alerts than you have to fully rebuild the state of machine applying incoming diffs to the last snapshot Currently it is done only for the list of installed packages used to track and check packages on servers over USN and NVD feeds

Alexander · Answer

We don t re sync Usually drops occurs during network problems or performance problems on logs collector We have queues to soften impact of short outage but cant do anything with long time problems So here we have to wait for the next snapshot

Alexander · Answer

Just to make it clear logs collector is not where osquery sends logs We have a service that gets logs from all osquerys batches them and sends to archive and logs collector That service is much more stable than our current collector

seph · Answer

You may want to look at the epoch incrementing functionality

Alexander · Answer

I see Can epoch be changed on some event inside osquery or only by config plugin

Zach Zeid · Answer

epoch incrementing functionality

Alexander · Answer

No sure if got it right Do you mean to use `schedule epoch` option and counter to rebuild state and detect drops

seph · Answer

epoch changes are triggered by the TLS server presumably also by the config plugin I m not aware of another way to change the epoch but I haven t gone digging too much

seph · Answer

Yes the intent of the epoch stuff is to facilitate resyncing the diffs You cannot correct sync snapshots and diffs that are separate queries You can kinda fake it if you only care about adds but there are dragons