Hi all I recently installed fleet 4 22 1 and created an rpm osquery #fleet

Hi all, I recently installed fleet 4.22.1 and crea...

Terje Kvernes

11/12/2022, 1:01 PM

Hi all, I recently installed fleet 4.22.1 and created an rpm that I tested on two clients. The UI itself works well and the clients report in, but when I try to run a query, I get:

Copy code

Nov 12 13:52:57 [...] fleet[79831]: {"component":"http","err":"read auth token: reading from websocket: sockjs: session not in open state","msg":"failed to read >
Nov 12 13:53:00 [...] fleet[79831]: {"component":"http","err":"error in query ingestion","ingestion-err":"campaign waiting for listener (please retry)","ip_addr">

The setup is fleet listening on localhost:8080 and nginx acting as a proxy to serve fleet on *:443. Serving the UI works well, and I have attempted to serve the API specifically via

Copy code

location ~/api/v1/osquery {
    grpc_pass <grpcs://127.0.0.1:8080>;
    grpc_set_header Host $host;
    grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_buffering off;
    access_log /var/log/nginx/api.fleetdm.com_access.log;
    error_log /var/log/nginx/api.fleetdm.com_error.log;
 }

And the access logs suggest everything works fine:

Copy code

2001:[...] - - [12/Nov/2022:13:58:57 +0100] "POST /api/v1/osquery/config HTTP/2.0" 200 472 "-" "osquery/5.5.1"
2001:[...] - - [12/Nov/2022:13:59:02 +0100] "POST /api/v1/osquery/distributed/read HTTP/2.0" 200 39 "-" "osquery/5.5.1"
2001:[...] - - [12/Nov/2022:13:59:06 +0100] "POST /api/v1/osquery/distributed/read HTTP/2.0" 200 39 "-" "osquery/5.5.1"

It is worth noting that the clients are dual stack and the infrastructure runs primarily over IPv6. I am not sure how to debug the report that websocket not being in "an open state". Is there anything in particular I should be looking for/at?

Benjamin Edwards

11/12/2022, 4:07 PM

I think there were some gotchas when it came to nginx and handling websocket connections properly. Check out https://mysteryincorporated.medium.com/nginx-configuration-for-fleetdm-setups-that-want-to-catch-3m-flatties-willem-powerfish-be-proud-7f99f97fdede by @Mystery Incorporated

Terje Kvernes

11/12/2022, 4:08 PM

Yeah, I saw that, that’s where I got the osquery grpc setup from. Hm. I’ll go through it again and see if I missed anything.

Benjamin Edwards

11/12/2022, 4:13 PM

Ahhh ok.

Mystery Incorporated

11/12/2022, 4:37 PM

i just tested and it works for me 4.22.1

Mystery Incorporated

11/12/2022, 4:39 PM

Copy code

location ~/{
    # Assuming your fleet server is listening on 8080
    proxy_pass <https://localhost:8080>;
    proxy_read_timeout 90;
    proxy_connect_timeout 90;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;                                                                    
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
    proxy_set_header Proxy "";
    access_log /var/log/nginx/ui.fleetdm.com_access.log;
    error_log /var/log/nginx/ui.fleetdm.com_error.log;
}

Mystery Incorporated

11/12/2022, 4:39 PM

Upgrade stuff is important on that bit too

Terje Kvernes

11/12/2022, 4:39 PM

Right. Let me check. 🙂

Terje Kvernes

11/12/2022, 4:43 PM

I think I’m going to trim down the nginx config and ensure I don’t have any modules loaded that cause issues.

Terje Kvernes

11/12/2022, 4:45 PM

I should probably also check how orbit is set up. A few too many moving parts right now, 🙂

Terje Kvernes

11/12/2022, 4:45 PM

Thanks!

Terje Kvernes

11/12/2022, 5:13 PM

…

Terje Kvernes

11/12/2022, 5:14 PM

I overwrote some of the proxy headers elsewhere. Trimming the config helped. Thanks!

Benjamin Edwards

11/12/2022, 5:40 PM

Working now?

Terje Kvernes

11/13/2022, 7:32 AM

Yep

Terje Kvernes

11/13/2022, 7:33 AM

Still seeing the public IP and private IP mixed up though, but that’s a different issue.

4 Views

Open in Slack

Previous Next