Hi all, I recently installed fleet 4.22.1 and crea...
# fleet
Hi all, I recently installed fleet 4.22.1 and created an rpm that I tested on two clients. The UI itself works well and the clients report in, but when I try to run a query, I get:
Copy code
Nov 12 13:52:57 [...] fleet[79831]: {"component":"http","err":"read auth token: reading from websocket: sockjs: session not in open state","msg":"failed to read >
Nov 12 13:53:00 [...] fleet[79831]: {"component":"http","err":"error in query ingestion","ingestion-err":"campaign waiting for listener (please retry)","ip_addr">
The setup is fleet listening on localhost:8080 and nginx acting as a proxy to serve fleet on *:443. Serving the UI works well, and I have attempted to serve the API specifically via
Copy code
location ~/api/v1/osquery {
    grpc_pass <grpcs://>;
    grpc_set_header Host $host;
    grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_buffering off;
    access_log /var/log/nginx/api.fleetdm.com_access.log;
    error_log /var/log/nginx/api.fleetdm.com_error.log;
And the access logs suggest everything works fine:
Copy code
2001:[...] - - [12/Nov/2022:13:58:57 +0100] "POST /api/v1/osquery/config HTTP/2.0" 200 472 "-" "osquery/5.5.1"
2001:[...] - - [12/Nov/2022:13:59:02 +0100] "POST /api/v1/osquery/distributed/read HTTP/2.0" 200 39 "-" "osquery/5.5.1"
2001:[...] - - [12/Nov/2022:13:59:06 +0100] "POST /api/v1/osquery/distributed/read HTTP/2.0" 200 39 "-" "osquery/5.5.1"
It is worth noting that the clients are dual stack and the infrastructure runs primarily over IPv6. I am not sure how to debug the report that websocket not being in "an open state". Is there anything in particular I should be looking for/at?
I think there were some gotchas when it came to nginx and handling websocket connections properly. Check out https://mysteryincorporated.medium.com/nginx-configuration-for-fleetdm-setups-that-want-to-catch-3m-flatties-willem-powerfish-be-proud-7f99f97fdede by @Mystery Incorporated
Yeah, I saw that, that’s where I got the osquery grpc setup from. Hm. I’ll go through it again and see if I missed anything.
Ahhh ok.
i just tested and it works for me 4.22.1
Copy code
location ~/{
    # Assuming your fleet server is listening on 8080
    proxy_pass <https://localhost:8080>;
    proxy_read_timeout 90;
    proxy_connect_timeout 90;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;                                                                    
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
    proxy_set_header Proxy "";
    access_log /var/log/nginx/ui.fleetdm.com_access.log;
    error_log /var/log/nginx/ui.fleetdm.com_error.log;
Upgrade stuff is important on that bit too
Right. Let me check. 🙂
I think I’m going to trim down the nginx config and ensure I don’t have any modules loaded that cause issues.
I should probably also check how orbit is set up. A few too many moving parts right now, 🙂
I overwrote some of the proxy headers elsewhere. Trimming the config helped. Thanks!
Working now?
Still seeing the public IP and private IP mixed up though, but that’s a different issue.