Title
#kolide
j

Jason W

01/28/2020, 6:30 PM
Hi - I have an odd message in my launcher logs (using the github kolide). Anyone have an idea of what might be causing this? Even just a point in the right direction would be fantastic.
{
  "caller": "query_target.go:75",
  "err": "fetching target memberships: rpc error: code = Unknown desc = OK: HTTP status code 200; transport: received the unexpected content-type \"text/html; charset=utf-8\"",
  "msg": "updating kolide_target_membership data",
  "severity": "error",
  "ts": "2020-01-28T18:28:29.977507Z"
}
s

seph

01/28/2020, 6:49 PM
kolide_target_membership
is a feature related an earlier version of our SaaS. It’s a debug level warning. I think it should be supressed in current launcher versions.
6:50 PM
Well… It’s a launcher feature that was only ever implemented in one of the kolide SaaS offerings. I don’t think we ever ported it to fleet.
j

Jason W

01/28/2020, 6:50 PM
interesting. i’ve recently rebuilt launcher from the .11.6 tag
6:52 PM
I can try a fresh checkout and build again I suppose…
s

seph

01/28/2020, 6:52 PM
Hrm. Clearly not supressed then.
6:52 PM
Regardless, it’s ignorable
j

Jason W

01/28/2020, 6:52 PM
yeah, I just have IT admins complaining (especially on windows) that fills up the event logs
s

seph

01/28/2020, 6:54 PM
Interesting. https://github.com/kolide/launcher/pull/416/files was supposed to move that to debug
j

Jason W

01/28/2020, 6:57 PM
looks like it did not, then
s

seph

01/28/2020, 6:57 PM
I think it did, but that the error is coming back somewhat differently, so the
if
isn’t firing
6:57 PM
If you fix it, PR welcome 😉
j

Jason W

01/28/2020, 6:58 PM
you know, I can follow go, but haven't taken the time to really learn. it would be garbage 🙂 I will poke around and perhaps file an issue for someone else to do a more proper PR
s

seph

01/28/2020, 6:59 PM
I’ll see if I can find time. It shouldn’t be hard to replicate
7:00 PM
Is this effecting platforms other than windows?
j

Jason W

01/28/2020, 7:00 PM
yeah, it’s on my mac as well. What I posed is from my mac. It’s just easier to ignore on the mac.
7:01 PM
haven’t checked linux, but I can.
s

seph

01/28/2020, 7:01 PM
Nah. mac is enough. Lemme see if I can re-produce quickly
j

Jason W

01/28/2020, 7:01 PM
thx
s

seph

01/28/2020, 7:16 PM
A quick test case here, does not reproduce this error.
j

Jason W

01/28/2020, 7:17 PM
crazy. ok, let me recompile launcher and build a binary and see what happens. Very odd.
7:17 PM
thanks for looking so fast!
s

seph

01/28/2020, 7:17 PM
And, as I look at your error more… You’re getting html back on an GRPC stream:
desc = OK: HTTP status code 200; transport: received the unexpected content-type \"text/html
7:17 PM
Does anything work? Are you running this with some http based load balancer?
j

Jason W

01/28/2020, 7:17 PM
oh, everything works
s

seph

01/28/2020, 7:18 PM
Still… It seems like you’re getting an grpc connection to an http endpoint.
j

Jason W

01/28/2020, 7:18 PM
it’s behind nginx acting as a proxy to restrict access to the admin interface.
7:18 PM
let me expose kolide “naked” for short bit and see if that goes away.
s

seph

01/28/2020, 7:20 PM
I’m not really sure how nginx handles this. but I strongly suspect it’s doing something weird. I know there’s been conversation about how to do load balancing and endpoint restrictions, but there are some gotchas/
7:21 PM
Yeah. testing without nginx seems like a good step. And digging though slack for conversations about ELB vs ALB might also help
j

Jason W

01/28/2020, 7:21 PM
yeah, I will probably move to something else (maybe traefik?) or just like you say, native AWS ALB
s

seph

01/28/2020, 7:22 PM
I don’t run fleet, nor have I tried to run grpc stuff in AWS. But I know I’ve seen it discussed here in #kolide
j

Jason W

01/28/2020, 7:23 PM
thanks! big big help
s

seph

01/28/2020, 7:23 PM
np!
j

Jason W

01/29/2020, 2:20 AM
thanks btw - just got a chance to really look, and found where I was proxying that grpc endpoint as http/1 and not grpc. all fixed.
s

seph

01/29/2020, 2:21 AM
Nice! Glad it worked out.
2:21 AM
TBH I’m surprise anything worked at all
j

Jason W

01/29/2020, 2:23 AM
my memory is really foggy on this, but I had found a few endpoints, the ones relevant here are
/kolide.agent.Api
and
kolide.launcher.QueryTarget
2:23 AM
it’s the latter that caused the errors not being proxied via grpc