Ryan
01/05/2022, 5:06 PMosqueryd worker (18095) stopping: Memory limits exceeded: 336852000
Tomas Touceda
01/05/2022, 5:25 PMRyan
01/05/2022, 5:26 PMTomas Touceda
01/05/2022, 5:27 PMRyan
01/05/2022, 5:30 PMwatch -n1 free -m
I can see that it 100MB per second of RAM is consumed, until the system runs out of RAM and everything gets killedzwass
01/05/2022, 5:35 PMRyan
01/05/2022, 5:35 PMzwass
01/05/2022, 5:36 PMIf you close the Fleet tab will the distributed query continue running in the background?No, it will stop running.
Ryan
01/05/2022, 5:36 PMzwass
01/05/2022, 5:38 PMIf the osquery watchdog kills the osquery worker, does Fleet try to issue the same query again when the worker restarts, or does it mark that as failed?Yes. osquery doesn't (currently) expose any mechanism for indicating that the watchdog killed a distributed query. @sharvil is going to be looking into improving the performance monitoring for live queries within osquery, maybe we can also look at detecting watchdog kills of queries?
Ryan
01/05/2022, 5:38 PMzwass
01/05/2022, 5:40 PMI’m also confused that it was allowed to consume so much memory by the osquery watchdog thereIIRC the watchdog checks the utilization on an interval so there may be a brief lag before it enforces those limits. For the highest level of production safety you may want to configure a max memory limit via cgroups.
Ryan
01/05/2022, 5:45 PMzwass
01/05/2022, 5:48 PMRyan
01/05/2022, 5:49 PM