Osquery uses basic SQL commands to leverage a relational data-model to describe a device.

osquery

Hi team, I’m willing to take participants into the development of fleet, where should I start with?

Hello! It depends on what kind of skills you have and how much time you'd like to invest. You can take a look at <https://github.com/fleetdm/fleet/issues> for ideas of things we would like to work on.

Random one for you - I tired this as a distributed query and it exceeded the memory limits of osqueryd:
```osqueryd worker (18095) stopping: Memory limits exceeded: 336852000```

furthermore, the kernel OOM killer got involved and started killing off important services such as Elasticsearch :grimacing:

seems to have also generated a huge amount of load on the hosts

there was a huge amount of iowait, and memory usage shot up so quickly our telemetry got killed before we could see, and the OOM killer started killing everything

Just realised this won’t be pinging anyone because it’s on an automated PR message, so CCing <@U0JFM04MS> and <@U02771P7WTU> as you’ve replied to me in other topics, in case you have some advice? :slightly_smiling_face:

hi there, TIL about <#C020FH7LGLB|fleet-dev> xD

is that CPU usage for fleet? or the host running the query?

that was on one of the hosts running the query yeah :slightly_smiling_face:

yeah, that's the tricky thing with osquery... what CPU/RAM did this host have?

I ran the distributed query over ~1400 servers, and the results came in I realised it was showing all instances Log4J across the estate, but it was showing all of them even the patched ones, so I think I just closed the tab.

Some questions:

1. If you close the Fleet tab will the distributed query continue running in the background? If so, can you connect back to it in a new browser tab to see the results, or cancel it?
2. If the osquery watchdog kills the osquery worker, does Fleet try to issue the same query again when the worker restarts, or does it mark that as failed?

We had 40 hosts which reacted badly to this query and the OOM killer killed the main service along with things like telegraf (telemetry) and SSH itself.

I can reproduce this - just tried it targeting a single host and with the command `watch -n1 free -m` I can see that it 100MB per second of RAM is consumed, until the system runs out of RAM and everything gets killed

:cold_sweat: It can be a relatively expensive query if there are a lot of JARs running on the system. That seems like more than I'd expect though.

&gt; If you close the Fleet tab will the distributed query continue running in the background?
No, it will stop running.

I’m also confused that it was allowed to consume so much memory by the osquery watchdog there

&gt; If the osquery watchdog kills the osquery worker, does Fleet try to issue the same query again when the worker restarts, or does it mark that as failed?
Yes. osquery doesn't (currently) expose any mechanism for indicating that the watchdog killed a distributed query. <@U0F2G55TM> is going to be looking into improving the performance monitoring for live queries within osquery, maybe we can also look at detecting watchdog kills of queries?

Ok cool, all good to understand :slightly_smiling_face:

&gt; I’m also confused that it was allowed to consume so much memory by the osquery watchdog there
IIRC the watchdog checks the utilization on an interval so there may be a brief lag before it enforces those limits. For the highest level of production safety you may want to configure a max memory limit via cgroups.

is there a way to do an ANALYZE on this query? I’d love to figure out what it is unhappy about

on its own, the first query runs absolutely fine

Approx how big is that list? The second query will then run a YARA scan on every one of those files.

Is it worth putting some sort of disclaimer on that query as being potentially heavy/disruptive to run do you think?