I'm having an issue with Fleet where, when searching for labels or hosts in the "Select Targets" field in a New Query, it takes an extremely long time to return results (~15-30 seconds, if at all). I have less than 300 hosts enrolled/online. I'm trying to troubleshoot and identify the bottleneck, but everything looks fine on the back end. Searches return results very quickly and seem complete. DB looks healthy and connections aren't maxed out. Hoping someone can point me in the right direction wrt troubleshooting.
(all running in Docker containers)
OS: Centos 7
Nginx load balancing reverse proxy: v1.14 (web sockets enabled)
Fleet: v2.4.0 (3 instances)
03/02/2020, 7:09 PM
Can you open your network tab in the browser web tools and see if any other requests are taking a long time? This is not an issue that is familiar to me.
03/02/2020, 7:53 PM
If I click the query icon on a host on the hosts page, the POST request to "targets" averaged 63ms.
When I typed the first six characters of the label name (or similar when it's a hostname) into the "Select Targets" input on the New Query page, it made a separate request for each character I typed and took between 33.04-34.26s for each request to return.
@zwass (or anyone else) based on the above, any ideas where I should be looking to find the bottle neck (Fleet/MySQL/Redis)? Everything else in the app seems quick and responsive, except selecting targets for queries and packs.
03/03/2020, 10:14 PM
It's a pretty bizarre issue... Are you able to get query stats from MySQL in some way? Slow query log?
03/05/2020, 3:56 PM
Slow query log includes this. Looks like this query: https://github.com/kolide/fleet/blob/master/server/datastore/mysql/hosts.go#L640
I don't know if it's helpful, but I have osquery running on some lxd containers that share a uuid with their host and other containers running on the host. I had to change host identifiers to 'instance' since I had multiple hosts getting enrolled in Fleet as the 'same' host.
Count: 11 Time=28.07s (308s) Lock=0.00s (0s) Rows=0.0 (0), kolide[kolide]@3hosts
SELECT DISTINCT *
id IN (
MATCH(host_name, uuid) AGAINST('S' IN BOOLEAN MODE)
id IN (
MATCH(ip_address) AGAINST('S' IN BOOLEAN MODE)
AND NOT deleted
Hi @zwass, forgot to "@" you in my original post so doing it now.