Title
#kolide
d

Display name

03/02/2020, 4:50 PM
I'm having an issue with Fleet where, when searching for labels or hosts in the "Select Targets" field in a New Query, it takes an extremely long time to return results (~15-30 seconds, if at all). I have less than 300 hosts enrolled/online. I'm trying to troubleshoot and identify the bottleneck, but everything looks fine on the back end. Searches return results very quickly and seem complete. DB looks healthy and connections aren't maxed out. Hoping someone can point me in the right direction wrt troubleshooting. (all running in Docker containers) OS: Centos 7 Nginx load balancing reverse proxy: v1.14 (web sockets enabled) Fleet: v2.4.0 (3 instances) MySQL: 8.0.19 Redis: v5.0.7
zwass

zwass

03/02/2020, 7:09 PM
Can you open your network tab in the browser web tools and see if any other requests are taking a long time? This is not an issue that is familiar to me.
d

Display name

03/02/2020, 7:53 PM
If I click the query icon on a host on the hosts page, the POST request to "targets" averaged 63ms. When I typed the first six characters of the label name (or similar when it's a hostname) into the "Select Targets" input on the New Query page, it made a separate request for each character I typed and took between 33.04-34.26s for each request to return.
9:58 PM
@zwass (or anyone else) based on the above, any ideas where I should be looking to find the bottle neck (Fleet/MySQL/Redis)? Everything else in the app seems quick and responsive, except selecting targets for queries and packs.
zwass

zwass

03/03/2020, 10:14 PM
It's a pretty bizarre issue... Are you able to get query stats from MySQL in some way? Slow query log?
d

Display name

03/05/2020, 3:56 PM
Slow query log includes this. Looks like this query: https://github.com/kolide/fleet/blob/master/server/datastore/mysql/hosts.go#L640 I don't know if it's helpful, but I have osquery running on some lxd containers that share a uuid with their host and other containers running on the host. I had to change host identifiers to 'instance' since I had multiple hosts getting enrolled in Fleet as the 'same' host. Count: 11 Time=28.07s (308s) Lock=0.00s (0s) Rows=0.0 (0), kolide[kolide]@3hosts  SELECT DISTINCT *  FROM hosts  WHERE  (  id IN (  SELECT id  FROM hosts  WHERE  MATCH(host_name, uuid) AGAINST('S' IN BOOLEAN MODE)  )  OR  id IN (  SELECT host_id  FROM network_interfaces  WHERE  MATCH(ip_address) AGAINST('S' IN BOOLEAN MODE)  )  )  AND NOT deleted  LIMIT N
9:17 PM
Hi @zwass, forgot to "@" you in my original post so doing it now.