Hi all, currently trying to get fleet working behi...
# kolide
Hi all, currently trying to get fleet working behind a TLS terminating load balancer on k8s. Saw on a couple of threads that tls should be set to false (as expected), but doing this, even if I port forward to localhost to test on
I get connection refused or “sending http request to https server”, and then using https, it works with a self-signed cert (I think the default one?). I checked my pod’s config, everything is mounting correctly, exec’d into the pod to check the config and it had
tls: false
fleet config_dump
with whatever flags you are using to see what Fleet actually picks up.
I just set the environment variable explicitly with:
Copy code
              value: "false"
and that worked, it's not storing what's in the config in the env vars it seems
Not sure if I understand your statement, but Fleet definitely does not write configuration to the environment variables.
https://github.com/kolide/fleet/blob/master/docs/infrastructure/configuring-the-fleet-binary.md Under the server_tls section, it has "config file format" and "environment variable". So in the case of your statement, it's not even using what's in the config file, it's resorting to the environment variables (where KOLIDE_SERVER_TLS is not set), and just defaulting to true
which is strange because it is used to read the creds to connect to mysql and redis
actually, correction, i read those creds from a secret and put them in environment variables, so perhaps the config was never being read correctly at all...
env vars take precedence over config file entries but not will not override if they are unset
Probably your config file syntax is incorrect and it's not being picked up.
Hmm so that seemed to be the problem, UI is all working behind the TLS terminating LB, on http, but after reading some other threads, seems like TLS has to be enabled in order for agents to enroll, or has this changed recently? can I terminate TLS at the LB and use http for the UI (working) and grpc for the agents? grpc seems to be the problem now, requiring tls to be enabled.
The agents must communicate with the server via TLS. It's fine to terminate TLS at the LB. Typical setup is
osqueryd <- HTTPS -> LB <- HTTP -> Fleet server
Ok cool that's exactly what I'm trying to do. My ingress looks like:
Copy code
apiVersion: <http://networking.k8s.io/v1beta1|networking.k8s.io/v1beta1>
kind: Ingress
  name: my-ingress
    <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
    - hosts:
        - <http://my.hostname.com|my.hostname.com>
      secretName: kolide-fleet-tls
    - host: <http://my.hostname.com|my.hostname.com>
          - backend:
              serviceName: kolide-fleet
              servicePort: 8080
Which has the UI working correctly. However, on an enroll of a launcher agent (with --insecure to avoid self-signed errors) I get this error in launcher logs:
Copy code
{"caller":"request_queries.go:133","err":"rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: authentication handshake failed: tls: first record does not look like a TLS handshake\"","method":"RequestQueries","reauth":false,"res":"null","severity":"info","took":"108.909µs","ts":"2020-06-18T03:04:08.961847Z","uuid":"9c800ad7-d6d9-4ce0-bd49-cf4d18cee684"
The relevant LB log lines are hex gibberish. Thanks for your assistance btw
Are you able to curl the endpoint over HTTPS? Sounds like you don't have HTTPS configured properly.
curling gives:
Copy code
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: <https://curl.haxx.se/docs/sslcerts.html>

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
But curling with --insecure flag retrieves the page successfully. If I port forward the fleet server and try to enroll via localhost:8080 (thus bypassing the LB), it throws the exact same error as in my prev message.
Some progress, looks like i got the tls terminating correctly, however get this from launcher now:
Copy code
  "caller": "extension.go:494",
  "err": "sending string logs: writing logs: transport error sending logs: rpc error: code = Unknown desc = OK: HTTP status code 200; transport: received the unexpected content-type \"text/html; charset=utf-8\"",
  "severity": "info",
  "ts": "2020-06-18T06:51:48.562111Z"
The host does not show as green in fleet, this seems to me I need some extra configuration in the LB to route these grpc calls...
Yeah sounds like your LB is serving HTML