In your situation I recommend setting a CPUQuota for the systemd service to limit osquery’s usage to something reasonable.

Thanks !! Does this work like CGroups, from the docs, it looks so.

I'm running into perf issues as well, is this the preferred method to constraining resources instead of

watchdog

?

I recommend cpuquota in addition to using the watchdog.

Hey @theopolis, In order to use a 20% cpu quota, there is a possibility that osquery is pegged at 20% of usage and cgroup counters would be happy. But then watchdog would also see a 100% usage right since it would be getting the view of the cgroup. In that case, the watchdog must run as either -1 or not run at all. This poses a situation, if the watchdog is running at -1, the memory limit is 10000M which can trigger an OOM killer. We would also have to apply a

CPUQuota

and

MemoryLimit

to the service and disable the watchdog. Which could cause the service to potentially exit. if the watchdog is running at any other limit, restrictive or normal, osquery would get respawned due to cgroup view of the cpu usage.

Is the implication that there'd be a race condition in which the osquery service would keep flapping because of how

watchdog

interacts with the osquery daemon service?

I understand the theory but I’m not sure this occurs in practice. I’ve run a pegged osquery by stressing a system with 200 process starts per second and the default watchdog never killed the process. The process did hover at 20% of a core as expected.

Are logs generated whenever watchdog kills an osqueryd process, or is that verbosity I need to enable in the config?

@Zach Zeid If you run osqueryd at

--verbose

you can see watchdog killing osquery.

Oct 06 05:09:23 <hostname> osqueryd[17003]: osqueryd worker (17769) stopping: Maximum sustainable CPU utilization limit exceeded: 12

ok, but it'll only show up if I pass

--verbose

in the flags file or something?

@theopolis If that’s the case, then it’s good.. PS: I was running osquery on a test box today and it dropped close to 15k logs per second.. I was watching the dropped count in the

auditctl -s

advance over a 10 minute period..

Yeap as you mentioned, there’s a lot of noise in audit and the real solution is merging the bpf approach and making sure that is stable.

would bpf be able to process the complex file paths logic in the limited 4096 instructions.. I am assuming that we are not changing this limit.

looking at

osquery_schedule

are the

_time

columns in nanoseconds?

https://github.com/osquery/osquery/blob/master/osquery/tables/utility/osquery.cpp#L251-L259 This might be a good place to start looking for it

looks like it's seconds? https://github.com/osquery/osquery/blob/6d57dc8066031b3859a8e1da0627740150d5a24d/osquery/core/sql/query_performance.h