Hi all! I deployed fleet on kubernetes, and I have a lot of hosts connected to it (about 19k). My DB is stacked with a lot of information because of this, and I noticed that I have A LOT of duplicate hosts in my DB: SELECT count(uuid) from hosts; returns 26342, and SELECT count(DISTINCT uuid) from hosts; (to get count of unique values) returns 19360. So there's about 7k duplicate hosts I belive. This causes my fleet web UI to be really slow. Any tips on how to manage this correctly and remove those duplicate hosts?

Hey @Ortal Kombat, I'm looking into this for you. Can you please let me know what version of Fleet you're running?

4.50.0

Are you using fleetd in your deployment?

No

Are you using

--host_identifier

in your osquery deployment? https://osquery.readthedocs.io/en/stable/installation/cli-flags/#:~:text=%2D%2Dhost_identifier%3Dhostname

Yes --host_identifier=instance

I think using

instance

is likely the cause of the duplicate records. Every time osquery is or the osquerydb is deleted, a new instance-unique uuid is generated and Fleet will treat it as a unique host record

Yes, I enroll VMs

I actually think with VMs in your environment, it might be best to leave the identifier set to

instance

. We want to avoid a a separate issue where VMs with the same hostname or UUID would fail to check in and would re-enroll over each other

No, actually most of them are physical computers. But a lot of them really didn't communicate with fleet in a while, so I think setting the host expiry will do help, I'll try that

Sounds good! Keep us posted

I set the host expiry to 21 days and it cleaned a really big portion of records from the hosts table. I also set the max activities age to 30 days. This really helped and my Fleet UI now runs much better. Thanks a lot for the help @Dale Ribeiro!

That's great, glad to hear it!