Hi everyone. We've noticed that our osquery clients are opening and tearing down connections each time they check in on the distributed interval. We have

tls_session_reuse: true

. Is that only for the logger perhaps? It's surprising to me that we have constant sessions up and down - this is currently 5000 per min and will go even more crazy as we scale.

we actually set it to false, some time ago, as it seemed to give us the best results... i didnt see any real reuse of the sessions

Thanks Juan.

@npamnani @Seshu Looks like this was an uptycs added feature. Any insight?

As far as I can see it seems that it only stores a single http client per thread, which is connected to only one endpoint, and if there are different endpoints to be contacted, then the connection has to be closed and reopened

It is supposed to help avoid connection churn. It also depends on how the receiving side is configured (protocol, keep-alive, duration etc.). If you are doing periodic distributed read, it doesn't help for that. It may be useful for logger, assuming there constant data (events etc) being sent

i see

NLB forwarding to Nginx? We have identical keepalive configuration with Nginx. What is your distributed_interval? We use 0 and we see 2 connections from Osquery to Nginx. Every time config kicks in, it creates another temporary connection that gets closed.

@Seshu our distributed interval was 10 sec, we changed to 60 as a test to see if connections dropped (which they did). Yes, NLB --> nginx --> fleet

Seconds between polling for new queries (default 60)

. So this is the sleep duration between sending distributed_read requests.

oh, does 0 mean it uses the default which is 60 seconds, or did you mean to type that your dist interval was set to 60? you have 0 above. that would be bad if it was hardcoded to 16 sec

It is kind'a confusing (I think), but it is the sleep duration between sending distributed requests. So we don't sleep between requests. By default, it sleeps for 60 seconds if you don't change the flag value.

./dispatcher/distributed_runner.cpp:      pause(std::chrono::seconds(FLAGS_distributed_interval));

oh, got it. thanks

I usually look for

Connection

header in request/response and also track

netstat

connection churn from Osquery.

right

🙂 In our config, we don't pause. We always have Osquery waiting for queries. We can send a query to thousands of hosts and get back answer in a second. If we pause, we will have to wait that amount of time to get responses back.

sure...that's why we set it to 10

@Seshu how many RPS does a single osquery instance generate with the interval set to 0? Do you use a strategy in which the server waits to respond to the request until there is a query ready?

interesting to think about. ya, you'd get a lot of request to the read api with that set to 0...as fast as it can do them? how would that even work?

Every distributed query waits for 12 odd seconds before the server sends back empty response. If one or more queries come through, we immediately send them to Osquery on the waiting socket. Osquery immediately sends another request without delay once it gets response from back-end. With keep-alive, it will be on the same connection. Like I mentioned above, it usually is 2 connections per host (1 for DR, 1 for Log). And an occasional config connection.

Ah, interesting strategy! Thanks for sharing 🙂

ya, that's slick. I like it

How does your agent behave when offline? Does it use a lot of CPU spamming distributed query connection attempts? Or is this something you changed in your agent?

It does exponential back-off...

Is that osquery functionality or something you changed?

I need to check what is in OSS and if it is same as what we have .. 😞

Ha no worries, I can check osquery. Thanks for the insights!

https://github.com/osquery/osquery/blob/master/osquery/remote/utility.h#L274 should be this

Ah nice, thanks Stefano. Makes sense we have that for all requests.

how many failed attempts until it starts backing off? Considering that our clients (currently) can only communicate to Fleet when on an end-user initiated VPN connection, we definitely don't have persistent connectivity. I don't think we have any problems when

distributed_interval

is set to 10 seconds, so I'm guessing it won't be an issue with 0. I'll have to test this out.

Hum so, the backoff happens immediately at the first failure, but the caveat is that the retry attempts for that specific request are configured by a flag that depends on where the request originated from. In the case of the distributed plugin, the default is 3 attempts total (

distributed_tls_max_attempts

) which means, 1 immediate, then if the previous failed, 1 after 1 second, 1 after 4 seconds. After this, the

distributed_interval

kicks in.

ah, thanks @Stefano Bonicatti. I should have realized as I'm familiar with the max attempts setting

@Dan Achin with Fleet you'll get a response to a distributed request pretty much immediately so you're going to have osquery making a lot of requests if you set the interval to 0.

Hmm, it sounds to me like that would actually be less though based on the info above: https://osquery.slack.com/archives/C08V7KTJB/p1619218588257100?thread_ts=1619213725.247500&cid=C08V7KTJB What am I missing?

Fleet doesn't wait to return an empty response.

oh...was he talking about something other than Fleet? i guess i assumed as I thought I was in the Fleet channel. 🙂

He was talking about the Uptycs commercial project which uses a fork of osquery.

K, thanks. Looks like we'll need to figure out why we see so many connections from clients to Fleet. The last count I had was around 300 per system and every check-in to the read endpoint is generating a new one.

I wonder if this could be a bug in the

tls_session_timeout

implementation? Would be good to file an issue with osquery if you can come up with a repro.

ah...i should have been looking at that setting. I could try setting it to 0 and allowing nginx to manage the sessions

Sorry. Haven't looked into it @Dan Achin. Don't have the OSS Osquery setup handy. Will try to get to it...

ok, thanks. let me know if you get to it, would appreciate it. I'll open a git issue today at some point.

Thanks. Will look into this tonight...

@Stefano Bonicatti - would appreciate your feedback on this. Thanks!