https://github.com/osquery/osquery logo
Title
d

Dan Achin

07/21/2021, 5:25 PM
Hello fleet. We are running into an issue where fleet is running out of open file. This seems to be a configuration within fleet itself as we've already tweaked limits.conf. I'm not familiar with Fleet's webserver, but is there a way to tweak open files for Fleet? We don't want to scale yet as we have plenty of other system resources.
Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 1s
t

Tomas Touceda

07/21/2021, 5:40 PM
Hello Dan, could you tell me a bit more about how you're running fleet?
d

Dan Achin

07/22/2021, 11:40 PM
Hey @Tomas Touceda, thanks for the reply. Turns out we missed a systemd setting to increase the open file limit for the process itself. Also, we found out that the root of the problem was that we were using T type instances in AWS which are burstable. We had a large group of systems try to register all at once and it caused the systems to burst, then run out of burstable credits. Once that happened, they barely processed anything, which led to DB connection stacking up and eventually running out of open files. We are all set now though. 🙂
t

Tomas Touceda

07/23/2021, 12:08 PM
thanks for all the details and glad you were able to sort it out!
👍 1