https://github.com/osquery/osquery logo
Powered by
Title
t

Terje Kvernes

11/12/2022, 1:01 PM
Hi all, I recently installed fleet 4.22.1 and created an rpm that I tested on two clients. The UI itself works well and the clients report in, but when I try to run a query, I get: 
Nov 12 13:52:57 [...] fleet[79831]: {"component":"http","err":"read auth token: reading from websocket: sockjs: session not in open state","msg":"failed to read >
Nov 12 13:53:00 [...] fleet[79831]: {"component":"http","err":"error in query ingestion","ingestion-err":"campaign waiting for listener (please retry)","ip_addr">
The setup is fleet listening on localhost:8080 and nginx acting as a proxy to serve fleet on *:443. Serving the UI works well, and I have attempted to serve the API specifically via 
location ~/api/v1/osquery {
    grpc_pass <grpcs://127.0.0.1:8080>;
    grpc_set_header Host $host;
    grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_buffering off;
    access_log /var/log/nginx/api.fleetdm.com_access.log;
    error_log /var/log/nginx/api.fleetdm.com_error.log;
 }
And the access logs suggest everything works fine: 
2001:[...] - - [12/Nov/2022:13:58:57 +0100] "POST /api/v1/osquery/config HTTP/2.0" 200 472 "-" "osquery/5.5.1"
2001:[...] - - [12/Nov/2022:13:59:02 +0100] "POST /api/v1/osquery/distributed/read HTTP/2.0" 200 39 "-" "osquery/5.5.1"
2001:[...] - - [12/Nov/2022:13:59:06 +0100] "POST /api/v1/osquery/distributed/read HTTP/2.0" 200 39 "-" "osquery/5.5.1"
It is worth noting that the clients are dual stack and the infrastructure runs primarily over IPv6. I am not sure how to debug the report that websocket not being in "an open state". Is there anything in particular I should be looking for/at?
b

Benjamin Edwards

11/12/2022, 4:07 PM
I think there were some gotchas when it came to nginx and handling websocket connections properly. Check out https://mysteryincorporated.medium.com/nginx-configuration-for-fleetdm-setups-that-want-to-catch-3m-flatties-willem-powerfish-be-proud-7f99f97fdede by @Mystery Incorporated
t

Terje Kvernes

11/12/2022, 4:08 PM
Yeah, I saw that, that’s where I got the osquery grpc setup from. Hm. I’ll go through it again and see if I missed anything.
b

Benjamin Edwards

11/12/2022, 4:13 PM
Ahhh ok.
m

Mystery Incorporated

11/12/2022, 4:37 PM
i just tested and it works for me 4.22.1
location ~/{
    # Assuming your fleet server is listening on 8080
    proxy_pass <https://localhost:8080>;
    proxy_read_timeout 90;
    proxy_connect_timeout 90;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;                                                                    
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
    proxy_set_header Proxy "";
    access_log /var/log/nginx/ui.fleetdm.com_access.log;
    error_log /var/log/nginx/ui.fleetdm.com_error.log;
}
Upgrade stuff is important on that bit too
t

Terje Kvernes

11/12/2022, 4:39 PM
Right. Let me check. 🙂
I think I’m going to trim down the nginx config and ensure I don’t have any modules loaded that cause issues.
I should probably also check how orbit is set up. A few too many moving parts right now, 🙂
Thanks!
…
I overwrote some of the proxy headers elsewhere. Trimming the config helped. Thanks!
b

Benjamin Edwards

11/12/2022, 5:40 PM
Working now?
t

Terje Kvernes

11/13/2022, 7:32 AM
Yep
Still seeing the public IP and private IP mixed up though, but that’s a different issue.
4 Views
#fleetJoin Slack
Powered by