Hey folks, we are encountering issues querying in ...
# kolided
Daniel Parry
06/19/2020, 1:18 PMHey folks, we are encountering issues querying in the fleet UI against ~5k hosts. We tend to get partial results or just the spinner. We're wondering if the issue is with websockets and our haproxy setup. Anyone have any experience of getting fleet working behind an haproxy LB? Any tips / tricks please? Thanks!
t
Tim
06/19/2020, 5:02 PMI'm having that same issue and opened a bug report a couple months ago. From what I can see the DB is maxed out to 100% when I run queries then it just sits and never returns all results. Haven't heard anything to fix this yet
d
defensivedepth
06/19/2020, 6:01 PM@Daniel Parry @Tim I believe this may help https://github.com/kolide/fleet/issues/2207
defensivedepth
06/19/2020, 6:02 PMd
Daniel Parry
06/19/2020, 9:37 PMIn our case, it feels maybe more like https://github.com/kolide/fleet/issues/1980
Daniel Parry
06/19/2020, 9:38 PMI am noticing maybe 25 deadlocks a minute in the logs. The suggestion there is to tune the
distributed_intervalconfig_refreshlogger_tls_periodDaniel Parry
06/19/2020, 9:40 PMI'm also not sure how the values that I see in fleetctl get options relate to the options set in the osqueryd flags. Much confusion 🙂
Daniel Parry
06/19/2020, 9:45 PMwell, https://osquery.readthedocs.io/en/stable/installation/cli-flags/#distributed-query-service-flags helps a bit, but gives no real guidance on tuning. Currently I have --distributed_interval=30 --config_refresh=300 and --logger_tls_period=300
Daniel Parry
06/21/2020, 1:48 PMOn further investigation, it seems that querying 5000 hosts via fleetctl works fine. When using firefox we sometimes get partial results and then the querying stalls and with chrome querying never works. This leads me to think that there is variation in the websockets libraries in use where the browser variations are exposing some weird interaction with haproxy that causes them to fail. Will look into it some more in the week!