any time I make a new label my DB writer instance blows up, and then my FleetDM instances run out of memory and die. This happens even when using a db.r5.24xlarge with 48 cores, 96 vCPU, and 768G memory. We have ~162000 hosts enrolled, so I'm guessing that's why the DB operation for adding a label is so intensive. Eventually the DB finishes the operation, and we can restart all our FleetDM instances, and things work again, but it's obviously not an optimal experience.

yeah, that's not ideal. also we need to tweak seen times

I added the label right after 19:40

do you add it through the UI?

yes

it goes through one code path, while the fleetctl one goes another

interesting. once the DB recovers from this label, I'll try the next one via fleetctl

well, from that second screenshot, the apply itself might not matter