Title
#fleet
Jocelyn Bothe

Jocelyn Bothe

09/15/2021, 7:52 PM
any time I make a new label my DB writer instance blows up, and then my FleetDM instances run out of memory and die. This happens even when using a db.r5.24xlarge with 48 cores, 96 vCPU, and 768G memory. We have ~162000 hosts enrolled, so I'm guessing that's why the DB operation for adding a label is so intensive. Eventually the DB finishes the operation, and we can restart all our FleetDM instances, and things work again, but it's obviously not an optimal experience.
😮 2
7:54 PM
7:57 PM
Tomas Touceda

Tomas Touceda

09/15/2021, 8:00 PM
yeah, that's not ideal. also we need to tweak seen times
8:01 PM
I'll investigate
Jocelyn Bothe

Jocelyn Bothe

09/15/2021, 8:01 PM
I added the label right after 19:40
Tomas Touceda

Tomas Touceda

09/15/2021, 8:02 PM
do you add it through the UI?
Jocelyn Bothe

Jocelyn Bothe

09/15/2021, 8:02 PM
yes
8:02 PM
does it make a difference?
Tomas Touceda

Tomas Touceda

09/15/2021, 8:03 PM
it goes through one code path, while the fleetctl one goes another
Jocelyn Bothe

Jocelyn Bothe

09/15/2021, 8:03 PM
interesting. once the DB recovers from this label, I'll try the next one via fleetctl
Tomas Touceda

Tomas Touceda

09/15/2021, 8:07 PM
well, from that second screenshot, the apply itself might not matter
8:12 PM
once the hosts get the labels, they return the value and that gets recorded. So we record 162k label executions. I'm guessing that's what's blowing up the db. We'll look into this