Hi, guys. In the seek of your help again: is there any way we can instruct the agents from Fleet to purge the RocksDB contents? Following with our tests we see a number of agents that have become unstable and we suspect it might be related to their internal database. Is there any way we could handle this cleanup centrally? Any advise or similar experiences?

Hi @Francisco Huerta. What do you mean when you use the phrase “become unstable?” Are these agents failing to communicate with Fleet?

Thanks, @Noah Talerman. We see errors from those agents trying to establish a connection to send data to the Fleet for example

Got it. Do you mind sharing the errors you’re seeing for those agents in this thread?

Thank you! I don’t have an immediate answer to your question on purging RocksDB via Fleet. Working on getting an answer for you now.

If you want to fully purge the rocksdb database you'd need to delete the database directory on the host. Can you run

--verbose --tls_dump

on one of those hosts experiencing the issue and see if you can tell what they are sending?

thanks both. yep, we're familiarized with purging the contents of the directory but we were looking at something automatic and/or centralized

This seems like a feature that could be useful to add to Orbit

For all intents and purposes, wouldn't setting

buffered_log_max=1

purge the local DBs? We dropped that setting way down when dealing with an issues were all POSTs from many of our clients ended up with http 400 errors through our nginx layer in front of Fleet. If you control your osquery options at Fleet, this is a quick way to get all of the local data to expire out. I've often wondered if there was a way to read the local rocks DB, or maybe a utility to check the integrity of the data?

I tried that parameter but that did not work neither

thanks, will look at sst_dump

osqueryd  --database_dump

.

buffered_log_max

will clear the buffered TLS logs, but there are other things stored in the DB

Is this a known osquery issue? I dont recall ever seeing that errror

Ok, so it seems we've got some stuff to look into: we've discovered some of the hosts that we are monitoring are particularly chatty when it comes to the generation of win events. A step by step description would be as follows:

interesting

thanks, @Dan Achin, good insights too