Ryan01/05/2022, 5:06 PM
osqueryd worker (18095) stopping: Memory limits exceeded: 336852000
Tomas Touceda01/05/2022, 5:25 PM
Ryan01/05/2022, 5:26 PM
Tomas Touceda01/05/2022, 5:27 PM
Ryan01/05/2022, 5:30 PM
I can see that it 100MB per second of RAM is consumed, until the system runs out of RAM and everything gets killed
watch -n1 free -m
zwass01/05/2022, 5:35 PM
Ryan01/05/2022, 5:35 PM
zwass01/05/2022, 5:36 PM
If you close the Fleet tab will the distributed query continue running in the background?No, it will stop running.
Ryan01/05/2022, 5:36 PM
zwass01/05/2022, 5:38 PM
If the osquery watchdog kills the osquery worker, does Fleet try to issue the same query again when the worker restarts, or does it mark that as failed?Yes. osquery doesn't (currently) expose any mechanism for indicating that the watchdog killed a distributed query. @sharvil is going to be looking into improving the performance monitoring for live queries within osquery, maybe we can also look at detecting watchdog kills of queries?
Ryan01/05/2022, 5:38 PM
zwass01/05/2022, 5:40 PM
I’m also confused that it was allowed to consume so much memory by the osquery watchdog thereIIRC the watchdog checks the utilization on an interval so there may be a brief lag before it enforces those limits. For the highest level of production safety you may want to configure a max memory limit via cgroups.
Ryan01/05/2022, 5:45 PM
zwass01/05/2022, 5:48 PM
Ryan01/05/2022, 5:49 PM