Channels
doorman
infrastructure
random
zercurity
community-feeds
fleet-dev
code-review
queryhub
apple-silicon
carving
tls
fim
goquery
zentral
aws
querycon
golang
zeek
file-carving
fuzzing
auditing-warroom
linen-dev
fleetosquery
plugins
jobs
arm-architecture
darkbytes
process-auditing
uptycs
android_tests
selfgroup
vendor-feeds
fleet
eclecticiq-polylogyx-extension
ebpf
website
core
general
macos
kolide
osctrl
extensions
foundation
sql
officehours
linux
windows
Title
j
Jocelyn Bothe
09/15/2021, 7:52 PMany time I make a new label my DB writer instance blows up, and then my FleetDM instances run out of memory and die. This happens even when using a db.r5.24xlarge with 48 cores, 96 vCPU, and 768G memory. We have ~162000 hosts enrolled, so I'm guessing that's why the DB operation for adding a label is so intensive. Eventually the DB finishes the operation, and we can restart all our FleetDM instances, and things work again, but it's obviously not an optimal experience.
😮 2
t
Tomas Touceda
09/15/2021, 8:00 PMyeah, that's not ideal. also we need to tweak seen times
I'll investigate
j
Jocelyn Bothe
09/15/2021, 8:01 PMI added the label right after 19:40
t
Tomas Touceda
09/15/2021, 8:02 PMdo you add it through the UI?
j
Jocelyn Bothe
09/15/2021, 8:02 PMyes
does it make a difference?
t
Tomas Touceda
09/15/2021, 8:03 PMit goes through one code path, while the fleetctl one goes another
j
Jocelyn Bothe
09/15/2021, 8:03 PMinteresting. once the DB recovers from this label, I'll try the next one via fleetctl
t
Tomas Touceda
09/15/2021, 8:07 PMwell, from that second screenshot, the apply itself might not matter
once the hosts get the labels, they return the value and that gets recorded. So we record 162k label executions. I'm guessing that's what's blowing up the db. We'll look into this