Hrm. I wonder what correct is. I understand that in this case moving to sigkill makes more sense. But what is the flip side? Does that make it more likely to corrupt the local db?

It might, although I would expect in many cases for the issues hitting the limit to have osquery being unresponsive to a nice stop anyway. If we had a bit less code, or had ingrained since the start a collaborative approach, between the table logic and the watchdog, then maybe having some kind of delay and expect the worker to exit cleanly was sort of ok. Another way would be to add two limits, so that there’s the usual limit that can be hit, we give a graceful kill, but if in the next 2 seconds or so the CPU consumption continue or the memory keeps increasing, then we give an immediate kill. This a tad less immediate to implement though.

Yes, agreed. In some ways, a 2 limit thing would make sense.

I would say that it should be the opposite. The machine osquery it runs on it’s more important than osquery, because osquery it’s not the reason for that machine to exist, there are other services.

On macOS there’s both ulimit, and something inside launchd.

I can only speak for us, but if osquery can affect services running, then we can’t run osquery. It’s as simple as that. The server being monitored is definitely more important than osquery itself (for us). We will run osquery on thousands of production servers and if we send a query to all of them that risks taking down even a fraction of them, we just cant run that query. Ideally, it should run the query where it can, and fail on the others.

Looping back to this, rather late, we talked about it a bit at office hours. I suspect we’re going to end up adding some configuration surface to it. Ultimately there’s no “right” answer, some sites will want to tune this to aggressive kill osquery, some will want to try to keep state.

Shameless plug, I’ve updated the PR with the separate delay