Fleet 2.6.0 / Osquery 4.4.0 on W10 with Launcher v0.11.11 I have a scheduled query for

windows_events

, but am getting the following errors in the W10 application logs:

caller=level.go:63 level=info caller=extension.go:494 err="sending string logs: writing logs: transport error sending logs: rpc error: code = Internal desc = grpc: error while marshaling: proto: field \"kolide.agent.LogCollection.Log.Data\" contains invalid UTF-8"

Any thoughts as to where to look next?

@seph is this the same UTF8 bug we've seen occasionally over quite some time?

I wasnt sure if this error was related to not getting windows_events logs or something else...

I think it could definitely be related to that

There are handful of utf8 issues in this ecosystem. Osquery sometimes screws up the encoding. This may be a rocksdb issue. There is some info about it at https://github.com/kolide/launcher/pull/481 some more at https://github.com/osquery/osquery/issues/5288

Looks like a major pr related to this was merged a couple weeks ago: https://github.com/osquery/osquery/pull/6338

Yeah, that’s the windows rework I alluded to. (thanks for finding the PR)

So that windows rework should make it more unlikely that osquery would emit bad utf8, but not impossible. And when it does, errors like this will show up and the scheduled query will be blocked until the offending log is removed from rocks... Is that accurate?

Probably. I haven’t thought about this in awhile, but that sounds right.

From an ops side, I would rather see the server (fleet) accept it, and then let me fix it within the parsing pipeline

Feel free to comment in any of the linked kolide github issues. this is unlikely to bubble up my list, but the comments would be noted

Will do

my guess is having fleet pass it along unmodified is relatively hard. (a bunch of the plumbing there is typed)_

hmmmmm

Not sure though. I don’t work much on fleet

@zwass thoughts?

I don't think the logs ever hit Fleet. It's an issue with gRPC encoding on the client side. See my comment https://github.com/kolide/launcher/issues/445#issuecomment-601867302

Question is what we should do. Launcher could attempt to repair, but we were dicy about that too

Oh

Heh I voted against it

I know 😛

I don't think I realized in that context that it prevents logs from sending entirely via Launcher

We might not have realized it at the time.

Given that information, I'd say go with this strategy but only for logs that fail send the first time with this error.

Want to comment? Using that error handling seems possible, but I’m not sure how yet, Haven’t read the code with that in mind

Do you prefer my comment on this issue or PR?

Probably not needed. I think I captured this sentiment in the PR commnt. If you have anything additional feel free to add it

Looks good. Thank you

Unclear. There are probably multiple places that cause this

So a temp fix is to clear rocks and restart launcher?