Hi all, currently trying to get fleet working behind a TLS terminating load balancer on k8s. Saw on a couple of threads that tls should be set to false (as expected), but doing this, even if I port forward to localhost to test on

<http://localhost:8080>

I get connection refused or “sending http request to https server”, and then using https, it works with a self-signed cert (I think the default one?). I checked my pod’s config, everything is mounting correctly, exec’d into the pod to check the config and it had

tls: false

Try

fleet config_dump

with whatever flags you are using to see what Fleet actually picks up.

I just set the environment variable explicitly with:

- name: KOLIDE_SERVER_TLS
              value: "false"

and that worked, it's not storing what's in the config in the env vars it seems

Not sure if I understand your statement, but Fleet definitely does not write configuration to the environment variables.

https://github.com/kolide/fleet/blob/master/docs/infrastructure/configuring-the-fleet-binary.md Under the server_tls section, it has "config file format" and "environment variable". So in the case of your statement, it's not even using what's in the config file, it's resorting to the environment variables (where KOLIDE_SERVER_TLS is not set), and just defaulting to true

env vars take precedence over config file entries but not will not override if they are unset

Hmm so that seemed to be the problem, UI is all working behind the TLS terminating LB, on http, but after reading some other threads, seems like TLS has to be enabled in order for agents to enroll, or has this changed recently? can I terminate TLS at the LB and use http for the UI (working) and grpc for the agents? grpc seems to be the problem now, requiring tls to be enabled.

The agents must communicate with the server via TLS. It's fine to terminate TLS at the LB. Typical setup is

osqueryd <- HTTPS -> LB <- HTTP -> Fleet server

.

Ok cool that's exactly what I'm trying to do. My ingress looks like:

apiVersion: <http://networking.k8s.io/v1beta1|networking.k8s.io/v1beta1>
kind: Ingress
metadata:
  name: my-ingress
  annotations:
    <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
spec:
  tls:
    - hosts:
        - <http://my.hostname.com|my.hostname.com>
      secretName: kolide-fleet-tls
  rules:
    - host: <http://my.hostname.com|my.hostname.com>
      http:
        paths:
          - backend:
              serviceName: kolide-fleet
              servicePort: 8080

Which has the UI working correctly. However, on an enroll of a launcher agent (with --insecure to avoid self-signed errors) I get this error in launcher logs:

{"caller":"request_queries.go:133","err":"rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: authentication handshake failed: tls: first record does not look like a TLS handshake\"","method":"RequestQueries","reauth":false,"res":"null","severity":"info","took":"108.909µs","ts":"2020-06-18T03:04:08.961847Z","uuid":"9c800ad7-d6d9-4ce0-bd49-cf4d18cee684"

The relevant LB log lines are hex gibberish. Thanks for your assistance btw

Are you able to curl the endpoint over HTTPS? Sounds like you don't have HTTPS configured properly.

curling gives:

curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: <https://curl.haxx.se/docs/sslcerts.html>

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

But curling with --insecure flag retrieves the page successfully. If I port forward the fleet server and try to enroll via localhost:8080 (thus bypassing the LB), it throws the exact same error as in my prev message.

Yeah sounds like your LB is serving HTML