If worker thread is executing a live query request and the CPU/memory exceeds the set limit, does watchdog thread kill the worker thread or the monitoring by watchdog thread is limited to scheduled queries only?

It will kill it unfortunately, the watcher process is very "dumb" so to speak.

re: watcher. I was thinking about adding perf tracing in there so that 1. we can tell whether it's stuck or just busy 2. save the stacktrace before we kill the worker of course this would only work on Linux. Thoughts? Bad idea?

That sounds amazing. I’m also curious if we can do a lot of good with a better systemd unit, which configures proper performance limits for osquery.

One issue I remember related to this is that the watcher may kill the worker, but then it will restart and retrieve the same live query from the server. There's no way for the server to know that it failed because of this.

In case of scheduled queries, when the worker thread picks up the interrupted /incomplete during last execution query , it blacklists that query ? Does that policy not apply for the live query ?

Correct