I'm having an issue with Fleet where, when searchi...
# kolided
Display name
03/02/2020, 4:50 PMI'm having an issue with Fleet where, when searching for labels or hosts in the "Select Targets" field in a New Query, it takes an extremely long time to return results (~15-30 seconds, if at all). I have less than 300 hosts enrolled/online. I'm trying to troubleshoot and identify the bottleneck, but everything looks fine on the back end. Searches return results very quickly and seem complete. DB looks healthy and connections aren't maxed out. Hoping someone can point me in the right direction wrt troubleshooting.
(all running in Docker containers)
OS: Centos 7
Nginx load balancing reverse proxy: v1.14 (web sockets enabled)
Fleet: v2.4.0 (3 instances)
MySQL: 8.0.19
Redis: v5.0.7
z
zwass
03/02/2020, 7:09 PM
Can you open your network tab in the browser web tools and see if any other requests are taking a long time? This is not an issue that is familiar to me.
d
Display name
03/02/2020, 7:53 PMIf I click the query icon on a host on the hosts page, the POST request to "targets" averaged 63ms.
When I typed the first six characters of the label name (or similar when it's a hostname) into the "Select Targets" input on the New Query page, it made a separate request for each character I typed and took between 33.04-34.26s for each request to return.
Display name
03/03/2020, 9:58 PM@zwass (or anyone else) based on the above, any ideas where I should be looking to find the bottle neck (Fleet/MySQL/Redis)? Everything else in the app seems quick and responsive, except selecting targets for queries and packs.
z
zwass
03/03/2020, 10:14 PM
It's a pretty bizarre issue... Are you able to get query stats from MySQL in some way? Slow query log?
d
Display name
03/05/2020, 3:56 PMSlow query log includes this. Looks like this query: https://github.com/kolide/fleet/blob/master/server/datastore/mysql/hosts.go#L640
I don't know if it's helpful, but I have osquery running on some lxd containers that share a uuid with their host and other containers running on the host. I had to change host identifiers to 'instance' since I had multiple hosts getting enrolled in Fleet as the 'same' host.
Count: 11 Time=28.07s (308s) Lock=0.00s (0s) Rows=0.0 (0), kolide[kolide]@3hosts
SELECT DISTINCT *
FROM hosts
WHERE
(
id IN (
SELECT id
FROM hosts
WHERE
MATCH(host_name, uuid) AGAINST('S' IN BOOLEAN MODE)
)
OR
id IN (
SELECT host_id
FROM network_interfaces
WHERE
MATCH(ip_address) AGAINST('S' IN BOOLEAN MODE)
)
)
AND NOT deleted
LIMIT N
Display name
03/20/2020, 9:17 PMHi @zwass, forgot to "@" you in my original post so doing it now.