Hi guys, Need help for setup of osquery. We have got a problem with a huge traffic to our infra when our endpoint /api/v1/logger is failed for any reasons. Does it have settings or flags for controlling failed retries attempts? We got a 1.5Gbit/sec traffic from 5k machines just for 1 hour and this load was increasing. Thanks in advance.

Hello @Denis, the logger mechanism is a bit different from the one of the config or distributed, so there is no separate retry mechanism, simply if the batch of logs has failed to be sent, in the next normal logger period will be sent again. The period is controlled by

--logger_tls_period

.

Hello @Stefano Bonicatti, thanks for answer. I read the docs regarding logger_tls_period and logger at all. And I didn't found solution this reason why I asked there) Clients received the 50x error only when full post request was sended.

When everything is working, what's the bandwidth used? If it's less than that, then it means that you have to tune the TLS logger to limit how much it sends each period, because you might be experiencing that the default seems ok, just because the log queue is emptied fast enough, but the default max limit is actually not ok for you.

When everything is working, what's the bandwidth used?

About 10-20Mbit/sec

there's also

--logger_tls_max_lines

that can help with the amount of data sent, which by default is

1024

.

Yes, this is true. logger_tls_max_lines is default. Could you please advice amount of logger_tls_max_lines for our cases, if it possible)

> Could you please advice amount of logger_tls_max_lines for our cases, if it possible) Well that's not possible for me to say. You have to evaluate that for your deployments. I would start from what's the bandwidth limit you're ready to support, and then go from there. With the period and the maximum amount of lines (plus the average size of each log line, which only you would know) you can calculate what's the theoretical average you would get if sending logs takes no time (beyond the fixed delay configured by the flag). You can also calculate what's the maximum, because there's a third flag which controls the maximum line size,

logger_tls_max_linesize

, which by default is 1MiB. But this might make less sense, because while it's true you could theorically get each line to be of that size, 1MiB is big (and unlikely I believe) Finally be careful to not backlog your clients, especially if you increase the period and/or reduce the amount of lines sent too much, because then the client RocksDB database will slowly grow and become slower. There's a limit even there,

buffered_log_max

, which if hit the DB will start dropping old logs. It's quite high currently (1M entries). Keep in mind though that dropping logs might also cause further slow downs, because the DB has to do work to do so, so this mechanism is only ok if the threshold being hit is only temporary.

Thank you so much for explanation!

Sorry I should correct myself, because I was thinking of the normal case, not the backlogged case. If you get backlogged, you can start accumulating snapshot results, and you then would send multiple "lines" which can be big. So while keeping a high number in

logger_tls_max_lines

is correct for events, can be problematic for snapshot queries. In hindsight osquery should've had a single flag with a bandwidth target to reach, and osquery addressing how much "lines" to fit in the batch automatically.