https://github.com/osquery/osquery logo
#fleet
Title
# fleet
j

Jocelyn Bothe

09/15/2021, 7:52 PM
any time I make a new label my DB writer instance blows up, and then my FleetDM instances run out of memory and die. This happens even when using a db.r5.24xlarge with 48 cores, 96 vCPU, and 768G memory. We have ~162000 hosts enrolled, so I'm guessing that's why the DB operation for adding a label is so intensive. Eventually the DB finishes the operation, and we can restart all our FleetDM instances, and things work again, but it's obviously not an optimal experience.
😮 2
t

Tomas Touceda

09/15/2021, 8:00 PM
yeah, that's not ideal. also we need to tweak seen times
I'll investigate
j

Jocelyn Bothe

09/15/2021, 8:01 PM
I added the label right after 19:40
t

Tomas Touceda

09/15/2021, 8:02 PM
do you add it through the UI?
j

Jocelyn Bothe

09/15/2021, 8:02 PM
yes
does it make a difference?
t

Tomas Touceda

09/15/2021, 8:03 PM
it goes through one code path, while the fleetctl one goes another
j

Jocelyn Bothe

09/15/2021, 8:03 PM
interesting. once the DB recovers from this label, I'll try the next one via fleetctl
t

Tomas Touceda

09/15/2021, 8:07 PM
well, from that second screenshot, the apply itself might not matter
once the hosts get the labels, they return the value and that gets recorded. So we record 162k label executions. I'm guessing that's what's blowing up the db. We'll look into this
5 Views