Hi everyone Hope anyone can provide some hints on a Fleet si

Question

Hi everyone Hope anyone can provide some hints on a Fleet sizing problem we re seeing in our labs We re running some stress tests with a single Fleet node and at a certain point we start seeing enrolling too often errors that lead to Fleet getting unstable Assuming there has to be a certain break point are there any techniques to prevent this problem e g enabling multiple network interfaces currently we only have one for osquery lt gt Fleet traffic Any config parameters to tweak

Francisco Huerta · Answer

As said any guidance similar experiences best practices would be very useful at this stage Thanks much

zwass · Answer

Are you running multiple instances of osquery on the same host to do this load testing

Francisco Huerta · Answer

hey < zwass> our setup is as follows 10x hosts running approx 500 dockers each for a total of 5 000 osquery instances Those 5 000 endpoints are hitting a single Fleet DM server eight core machine

Francisco Huerta · Answer

we got two type of errors enrolling too often and also TLS handshake error EOF

Francisco Huerta · Answer

mySQL is running on a separate VM configured with a max of 400 simultaneous connections

zwass · Answer

EOF could be due to running out of open sockets file descriptors and might require adjusting `ulimit` on the server and or docker hosts

Francisco Huerta · Answer

that s something we suspected too but we increased the `ulimit` parameter and at peak moments we are not close to that limit

zwass · Answer

What are you setting for ` host identifier` in the Docker hosts

zwass · Answer

Depending on the deployment scenario that is often a cause of enrolling too often

zwass · Answer

Setting it to `instance` tends to help

Francisco Huerta · Answer

we are not setting it so I guess it must be getting the default value

Francisco Huerta · Answer

I cannot see the `instance` in the documentation are you meaning setting it as ` host identifier = instance`

Francisco Huerta · Answer

what would this be helpful for

Francisco Huerta · Answer

appreciate all prompt replies by the way thanks +1

zwass · Answer

In case the containers are sharing hardware UUIDs this helps Fleet see each container as a separate instance of osquery

Francisco Huerta · Answer

got you we will give it a try thanks so much

Francisco Huerta · Answer

sorry < zwass> do yoy mean ` host identifier = uuid` or is it ` host identifier = instance` just to confirm I m doing it right

zwass · Answer

Try using instance

Francisco Huerta · Answer

+1

Francisco Huerta · Answer

Hey < zwass> As an update we ve been testing the performance with host identifier = instance and we don t see much of difference After a certain threshold we see again EOF messages popping up

Francisco Huerta · Answer

When this happens we see an increase in the number of database connections from a flat average of 50 when everything works fine to peaks of 400 our limit

Francisco Huerta · Answer

CPU consumption also gets increased to 100

Francisco Huerta · Answer

we ve tried creating a second network interface to balance incoming connections from the agents to the Fleet manager but we don t see a significant improvement here either

zwass · Answer

This is CPU consumption on the Fleet server or the MySQL server

zwass · Answer

FWIW I ve load tested Fleet to 150 000+ simulated devices and folks are using Fleet in production on close to 100 000 devices

zwass · Answer

At around 5 000 devices you might want to think about adding a load balancer routing traffic to multiple Fleet servers But I know I can get more than that running on just my mac laptop

zwass · Answer

The TLS EOF errors are in the osquery logs or the Fleet server logs

zwass · Answer

As for the enrolling too often error I found a bug in osquery that is probably causing this <https github com osquery osquery issues 6993> Because of that we are disabling the enrollment cooldown by default in the next release coming out today Pulling that down once we release it could help address that issue

Francisco Huerta · Answer

Thanks trying to answer your questions comments in order

Francisco Huerta · Answer

CPU consumption refers to the Fleet server

Francisco Huerta · Answer

Yes we ve got a load balancer in front of the server

Francisco Huerta · Answer

TLS EOF are reported by the Fleet server

Francisco Huerta · Answer

thanks for the indication on the bug we will look into it +1

Francisco Huerta · Answer

An extra insight we see improvements when setting tls session reuse = false

Francisco Huerta · Answer

it seems that in our case and maybe due to the way we create simulated endpoints a number of connections are kept open over time causing Fleet eventually to get unstable