Channels
doorman
infrastructure
random
zercurity
community-feeds
fleet-dev
code-review
queryhub
apple-silicon
carving
tls
fim
goquery
zentral
aws
querycon
golang
zeek
file-carving
fuzzing
auditing-warroom
linen-dev
fleetosquery
plugins
jobs
arm-architecture
darkbytes
process-auditing
uptycs
android_tests
selfgroup
vendor-feeds
fleet
eclecticiq-polylogyx-extension
ebpf
website
core
general
macos
kolide
osctrl
extensions
foundation
sql
officehours
linux
windows
Title
d
Daniel Parry
06/19/2020, 1:18 PMHey folks, we are encountering issues querying in the fleet UI against ~5k hosts. We tend to get partial results or just the spinner. We're wondering if the issue is with websockets and our haproxy setup. Anyone have any experience of getting fleet working behind an haproxy LB? Any tips / tricks please? Thanks!
t
Tim
06/19/2020, 5:02 PMI'm having that same issue and opened a bug report a couple months ago. From what I can see the DB is maxed out to 100% when I run queries then it just sits and never returns all results. Haven't heard anything to fix this yet
d
defensivedepth
06/19/2020, 6:01 PM@Daniel Parry @Tim I believe this may help https://github.com/kolide/fleet/issues/2207
d
Daniel Parry
06/19/2020, 9:37 PMIn our case, it feels maybe more like https://github.com/kolide/fleet/issues/1980
I am noticing maybe 25 deadlocks a minute in the logs. The suggestion there is to tune the
distributed_intervalconfig_refreshlogger_tls_periodI'm also not sure how the values that I see in fleetctl get options relate to the options set in the osqueryd flags. Much confusion 🙂
well, https://osquery.readthedocs.io/en/stable/installation/cli-flags/#distributed-query-service-flags helps a bit, but gives no real guidance on tuning. Currently I have --distributed_interval=30 --config_refresh=300 and --logger_tls_period=300
On further investigation, it seems that querying 5000 hosts via fleetctl works fine. When using firefox we sometimes get partial results and then the querying stalls and with chrome querying never works. This leads me to think that there is variation in the websockets libraries in use where the browser variations are exposing some weird interaction with haproxy that causes them to fail. Will look into it some more in the week!