any time I make a new label my DB writer instance ...
# fleetj
Jocelyn Bothe
09/15/2021, 7:52 PMany time I make a new label my DB writer instance blows up, and then my FleetDM instances run out of memory and die. This happens even when using a db.r5.24xlarge with 48 cores, 96 vCPU, and 768G memory. We have ~162000 hosts enrolled, so I'm guessing that's why the DB operation for adding a label is so intensive. Eventually the DB finishes the operation, and we can restart all our FleetDM instances, and things work again, but it's obviously not an optimal experience.
😮 2
Jocelyn Bothe
09/15/2021, 7:54 PMJocelyn Bothe
09/15/2021, 7:57 PMt
Tomas Touceda
09/15/2021, 8:00 PMyeah, that's not ideal. also we need to tweak seen times
Tomas Touceda
09/15/2021, 8:01 PMI'll investigate
j
Jocelyn Bothe
09/15/2021, 8:01 PMI added the label right after 19:40
t
Tomas Touceda
09/15/2021, 8:02 PMdo you add it through the UI?
j
Jocelyn Bothe
09/15/2021, 8:02 PMyes
Jocelyn Bothe
09/15/2021, 8:02 PMdoes it make a difference?
t
Tomas Touceda
09/15/2021, 8:03 PMit goes through one code path, while the fleetctl one goes another
j
Jocelyn Bothe
09/15/2021, 8:03 PMinteresting. once the DB recovers from this label, I'll try the next one via fleetctl
t
Tomas Touceda
09/15/2021, 8:07 PMwell, from that second screenshot, the apply itself might not matter
Tomas Touceda
09/15/2021, 8:12 PMonce the hosts get the labels, they return the value and that gets recorded. So we record 162k label executions. I'm guessing that's what's blowing up the db. We'll look into this
5 Views