Title
#kolide
d

Daniel Parry

06/19/2020, 1:18 PM
Hey folks, we are encountering issues querying in the fleet UI against ~5k hosts. We tend to get partial results or just the spinner. We're wondering if the issue is with websockets and our haproxy setup. Anyone have any experience of getting fleet working behind an haproxy LB? Any tips / tricks please? Thanks!
Tim

Tim

06/19/2020, 5:02 PM
I'm having that same issue and opened a bug report a couple months ago. From what I can see the DB is maxed out to 100% when I run queries then it just sits and never returns all results. Haven't heard anything to fix this yet
defensivedepth

defensivedepth

06/19/2020, 6:01 PM
@Daniel Parry @Tim I believe this may help https://github.com/kolide/fleet/issues/2207
d

Daniel Parry

06/19/2020, 9:37 PM
In our case, it feels maybe more like https://github.com/kolide/fleet/issues/1980
9:38 PM
I am noticing maybe 25 deadlocks a minute in the logs. The suggestion there is to tune the
distributed_interval
,
config_refresh
, and
logger_tls_period
flags but I'm not exactly sure what values are good, and have not found a good definition yet of what those settings actually do.
9:40 PM
I'm also not sure how the values that I see in fleetctl get options relate to the options set in the osqueryd flags. Much confusion 🙂
9:45 PM
well, https://osquery.readthedocs.io/en/stable/installation/cli-flags/#distributed-query-service-flags helps a bit, but gives no real guidance on tuning. Currently I have --distributed_interval=30 --config_refresh=300 and --logger_tls_period=300
1:48 PM
On further investigation, it seems that querying 5000 hosts via fleetctl works fine. When using firefox we sometimes get partial results and then the querying stalls and with chrome querying never works. This leads me to think that there is variation in the websockets libraries in use where the browser variations are exposing some weird interaction with haproxy that causes them to fail. Will look into it some more in the week!