https://github.com/osquery/osquery logo
Powered by
Title
z

zwass

01/13/2021, 7:46 PM
Launcher communicates over GRPC which uses HTTP/2 and is not always well supported by load balancers. Could this be the issue?
j

Justin Bowen

01/13/2021, 8:03 PM
https://aws.amazon.com/blogs/aws/new-application-load-balancer-support-for-end-to-end-http-2-and-grpc/
https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-http2
Figured id plop those here as I have ran into that issue before
:ty: 1
z

zwass

01/13/2021, 8:25 PM
@Carlo Miguel Cruz
c

Carlo Miguel Cruz

01/13/2021, 8:41 PM
Thanks for the info. We also tried using AWS ALB for HTTP2 and also with gRPC but we get the same error (
err":"enrolling host: transport error in enrollment: rpc error: code = Unavailable desc = unavailable"
) from launcher. The fleetdm web console also gets 
HTTP error 502 Bad Gateway
on HTTP2. The web console gets 
HTTP error 464
when we set the target group to gRPC. That is why we went back to using NLB.
We bumped fleetdm from 3.5.1 to 3.6.0 but we still get the same error.
z

zwass

01/13/2021, 10:00 PM
This seems almost certainly an LB problem
c

Carlo Miguel Cruz

01/14/2021, 5:20 AM
Most certainly that this is caused by the load balancer. We want to learn what network situations can cause the 
rpc error: code = Unavailable desc = unavailable
so we can act on it. We are still investigating as well. Were there similar reports like this before?
Hi guys, we were only able to fix this issue by terminating TLS on the fleetdm server instead of the load balancer for the gRPC connections. I think this has to be documented as a requirement for fleetdm to be able to receive gRPC connections from launcher. Using 
--insecure --insecure_transport
still did not allow gRPC to connect directly to the fleetdm server. So what we did for now is to terminate SSL on the application load balancer for the UI using AWS ACM. We used a separate network load balancer for the gRPC connection and just terminated the SSL directly on the fleetdm pods using a self-signed certificate. We have separate endpoints now for accessing the UI and for the gRPC connections. Thanks for helping us. I hope you may find our feedback useful as well. All the best!
2 Views
#fleetJoin Slack
Powered by