We're having to roll back FleetDM to an earlier version (much earlier). The detail updates put so much load on our DB it becomes non-functional, even with our actual queries completely turned off. We're using the largest RDS instance AWS has, and it is just completely crushed. It's so bad, we're going to have to start exploring other central management solutions. 😞

Which version where you on and which version did you roll back to?

hi Jocelyn, I'm sorry to hear you're having so much trouble. Based on our previous conversation, we are going to be implementing a worker style architecture to have more control over how many instances write to the db, which will give setups like yours much finer grained control. We will try to have this ready for 4.4.0 which is expected to be released in the beginning of October

we were on 4.3.0 and we're rolling back to 3.6.0.

@Jocelyn Bothe can you confirm that 3.6.0 is stable? If so, we can look at what changed between those versions and address any issues that we identify.

yup

--config_refresh=600
--config_accelerated_refresh=60
--distributed_interval=60

makes sense. The plan is to allow for much more granular control over writes, which will become eventually consistent rather than how it is today if you use it, but it will reduce the load on the writer

In the meantime, you could also make the Fleet

label_update_interval

and

detail_update_interval

longer so that fewer writes are generated for those top two queries you list.

label update was set to 120m and detail_update was set to 1440m

We're going to push a couple more improvements to label performance in the 4.3.1 release which we will get out by the middle of this week -- hopefully they will help.