Hey all, Just updated fleet to 4.17.0, and having a problem with distributed queries. Im getting the following error:

{
  "component": "http",
  "err": "error in query ingestion",
  "ingestion-err": "campaign waiting for listener (please retry)",
  "ip_addr": "ENDPOINT-IP:41730",
  "level": "error",
  "method": "POST",
  "took": "1.136788ms",
  "ts": "2022-07-12T20:38:50.060101107Z",
  "uri": "/api/v1/osquery/distributed/write",
  "x_for_ip_addr": ""
}

Also getting (although not sure its related):

{
  "component": "http",
  "err": "read auth token: reading from websocket: sockjs: session not in open state",
  "msg": "failed to read auth token",
  "ts": "2022-07-12T20:37:57.77330272Z"
}

The problem appears to be the agent talking back to the fleet server, because I can see the query being run on the agent in debug mode. It just seems to fail when posting back the results. Agent is vanilla OSquery 5.1.0 This only started since I updated a few minutes ago from fleet 4.9.1

Moving the conversation into a thread. Sorry about that! Are you using a proxy?

Nope

There were some changes in 4.13.2 related to websockets. It seems like we're starting to see these issues when websocket traffic isn't allowed because SockJS isn't reliable as a workaround. Can you tell me more about your Fleet/MySQL/Redis setup? I can reach out to the team and see what they suggest we poke at.

all on a single server spun up with docker-compose

Would you mind sharing your

compose

file?

I dm'd it to you

Erm, i'm not sure if this is related or not, but we updated to 4.1.7 yesterday and all our hosts show as offline in fleet.

@Jason Cetina Can you check your logs and see if you're seeing similar errors?

@Jason Cetina what version did you upgrade from? We have seen some issues with load balancers using deprecated SSL configurations that have been removed from support in the Go stdlib HTTP libraries we use.

we upgraded from 4.1.6

When you say 4.1.7 and 4.1.6, do you mean 4.17.0 and 4.16.0?

orry i'm a dummy yes 4.17

@Ari Weinberg can you open your browser devtools on the live query page and see if you are getting any errors in the network tab or the JS console?

@Kathy Satterlee fleet logs or osquery logs? do we need debugging set to true?

@Jason Cetina can you look at your Fleet server logs for any errors?

@zwass do I need debugging on or no?

should not need it, but let's see what is in your logs

just to be clear, the weird part is that this is only for part of our fleet

that sounds very likely to be related to the LB configuration

i think so, too. I will dig again.

Glad to hear it!

You don't have any kind of load balancer or anything? Looks like the LB websockets issue we commonly see.

I dont think so

So you have Fleet running on a server with docker-compose -- are you connecting directly to that server?

Where would the proxy cause an issue? between the server and agent? or between the client (web UI) and server?

web UI and server

Ahhh. There is a proxy there

Typically googling "<name of proxy> websocket configuration" is the best way to address this

Will do. Thanks so much!!!!

Thanks for the assist, @zwass !

Confirmed that's the problem by bypassing the proxy and going directly to the server. @zwass @Kathy Satterlee Thanks so much for all your help!

Glad to hear it!

Awesome news!

@Kathy Satterlee @zwass - just to close the loop here, we didn't have osqueryd configured to use the OS maintained cert bundle in

/etc/ssl/certs/ca-certificates.crt

. The root CA changed for this endpoint and so everything turfed when it got rotated. Not sure how/why it was setup that way. Anyway, it's fixed now.

Sweet! Thank you 🙂

Again, sorry for the noise.

Thanks for the info! And for reaching out :)

Sooooo, hate to re-open this, but I fixed the websocket issue, and I'm still getting no results... I'm getting the same error on the fleet server as before:

{
  "component": "http",
  "err": "error in query ingestion",
  "ingestion-err": "campaign stopped",
  "ip_addr": "AGENT-IP:50524",
  "level": "error",
  "method": "POST",
  "took": "2.406993ms",
  "ts": "2022-07-15T15:11:55.292524773Z",
  "uri": "/api/v1/osquery/distributed/write",
  "x_for_ip_addr": ""
}

Getting the following in the console: