Title
#kolide
k

Kyle

06/17/2020, 2:03 AM
Hi all, currently trying to get fleet working behind a TLS terminating load balancer on k8s. Saw on a couple of threads that tls should be set to false (as expected), but doing this, even if I port forward to localhost to test on
<http://localhost:8080>
I get connection refused or “sending http request to https server”, and then using https, it works with a self-signed cert (I think the default one?). I checked my pod’s config, everything is mounting correctly, exec’d into the pod to check the config and it had
tls: false
zwass

zwass

06/17/2020, 3:13 AM
Try
fleet config_dump
with whatever flags you are using to see what Fleet actually picks up.
k

Kyle

06/17/2020, 3:17 AM
I just set the environment variable explicitly with:
- name: KOLIDE_SERVER_TLS
              value: "false"
and that worked, it's not storing what's in the config in the env vars it seems
zwass

zwass

06/17/2020, 3:19 AM
Not sure if I understand your statement, but Fleet definitely does not write configuration to the environment variables.
k

Kyle

06/17/2020, 3:21 AM
https://github.com/kolide/fleet/blob/master/docs/infrastructure/configuring-the-fleet-binary.md Under the server_tls section, it has "config file format" and "environment variable". So in the case of your statement, it's not even using what's in the config file, it's resorting to the environment variables (where KOLIDE_SERVER_TLS is not set), and just defaulting to true
3:22 AM
which is strange because it is used to read the creds to connect to mysql and redis
3:23 AM
actually, correction, i read those creds from a secret and put them in environment variables, so perhaps the config was never being read correctly at all...
zwass

zwass

06/17/2020, 4:16 AM
env vars take precedence over config file entries but not will not override if they are unset
4:16 AM
Probably your config file syntax is incorrect and it's not being picked up.
k

Kyle

06/18/2020, 12:39 AM
Hmm so that seemed to be the problem, UI is all working behind the TLS terminating LB, on http, but after reading some other threads, seems like TLS has to be enabled in order for agents to enroll, or has this changed recently? can I terminate TLS at the LB and use http for the UI (working) and grpc for the agents? grpc seems to be the problem now, requiring tls to be enabled.
zwass

zwass

06/18/2020, 2:42 AM
The agents must communicate with the server via TLS. It's fine to terminate TLS at the LB. Typical setup is
osqueryd <- HTTPS -> LB <- HTTP -> Fleet server
.
k

Kyle

06/18/2020, 3:12 AM
Ok cool that's exactly what I'm trying to do. My ingress looks like:
apiVersion: <http://networking.k8s.io/v1beta1|networking.k8s.io/v1beta1>
kind: Ingress
metadata:
  name: my-ingress
  annotations:
    <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
spec:
  tls:
    - hosts:
        - <http://my.hostname.com|my.hostname.com>
      secretName: kolide-fleet-tls
  rules:
    - host: <http://my.hostname.com|my.hostname.com>
      http:
        paths:
          - backend:
              serviceName: kolide-fleet
              servicePort: 8080
Which has the UI working correctly. However, on an enroll of a launcher agent (with --insecure to avoid self-signed errors) I get this error in launcher logs:
{"caller":"request_queries.go:133","err":"rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: authentication handshake failed: tls: first record does not look like a TLS handshake\"","method":"RequestQueries","reauth":false,"res":"null","severity":"info","took":"108.909µs","ts":"2020-06-18T03:04:08.961847Z","uuid":"9c800ad7-d6d9-4ce0-bd49-cf4d18cee684"
The relevant LB log lines are hex gibberish. Thanks for your assistance btw
zwass

zwass

06/18/2020, 3:19 AM
Are you able to curl the endpoint over HTTPS? Sounds like you don't have HTTPS configured properly.
k

Kyle

06/18/2020, 4:03 AM
curling gives:
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: <https://curl.haxx.se/docs/sslcerts.html>

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
But curling with --insecure flag retrieves the page successfully. If I port forward the fleet server and try to enroll via localhost:8080 (thus bypassing the LB), it throws the exact same error as in my prev message.
6:56 AM
Some progress, looks like i got the tls terminating correctly, however get this from launcher now:
{
  "caller": "extension.go:494",
  "err": "sending string logs: writing logs: transport error sending logs: rpc error: code = Unknown desc = OK: HTTP status code 200; transport: received the unexpected content-type \"text/html; charset=utf-8\"",
  "severity": "info",
  "ts": "2020-06-18T06:51:48.562111Z"
}
The host does not show as green in fleet, this seems to me I need some extra configuration in the LB to route these grpc calls...
zwass

zwass

06/18/2020, 4:06 PM
Yeah sounds like your LB is serving HTML