Title
#linux
a

Artemis Tosini

08/18/2022, 7:13 PM
I'm looking at adding some containerd tables. Unfortunately the API is via grpc so we'd require grpc as a dependancy and either pregenerate the C++ with protoc or use protoc at build time, does anyone have opinions?
zwass

zwass

08/18/2022, 7:19 PM
I think it's reasonable to use the grpc API (it's connecting only over a unix domain socket, not a true remote API). Between pregenerate and build time, I don't have much preference... Whatever is cleaner/easier sgtm.
7:19 PM
cc @alessandrogario @Stefano Bonicatti
a

alessandrogario

08/18/2022, 7:21 PM
I seem to recall what was the hiccup, I think the grpc library is rather heavy and it comes with several additional libraries
7:21 PM
They would all need to be converted to CMake
7:23 PM
Going by memory: at least absl, cares, protobuf and probably a couple of additional ones
7:23 PM
7:23 PM
They have a 22k lines CMakeLists.txt file
7:27 PM
We have an impl of that in our https://github.com/trailofbits/osquery fork, though it's from 2020 and was no longer updated
7:28 PM
I think a PR was never made because the amount of code and dependencies it added was quite big
zwass

zwass

08/18/2022, 7:28 PM
Are you opposed to adding it? Or just warning that it's nontrivial?
a

alessandrogario

08/18/2022, 7:30 PM
I think it's going to generate some maintenance work if we want to do it properly
7:31 PM
compared to C++, golang could implement this more easily
7:32 PM
I think the major fault here is the grpc C++ SDK which is quite messy and even lacks features (async was only half-implemented)
s

seph

08/18/2022, 7:32 PM
I suspect that the go side still pulls in a lot of libs, but the build system hides that mess from you.
a

Artemis Tosini

08/18/2022, 7:33 PM
Yeah, that's amost certainly true. It's probably still pulling in protoc
zwass

zwass

08/18/2022, 7:33 PM
We could definitely build these tables as a Go extension, but I suspect folks might be disappointed it doesn't go into core osquery?
7:33 PM
Or, how crazy would it be to try to build a library in Go and then static link and call that from osquery?
a

Artemis Tosini

08/18/2022, 7:34 PM
I'm not sure how much Go would like that, it needs to run its gc
zwass

zwass

08/18/2022, 7:34 PM
That would open the floodgates to tons of other tables being built in Go, which would certainly make building osquery core tables much more accessible.
a

alessandrogario

08/18/2022, 7:34 PM
I think it depends on how much cgo is present in there
zwass

zwass

08/18/2022, 7:34 PM
Cursory searching indicates Go can be called from C++ but I have no idea how that works with GC.
a

alessandrogario

08/18/2022, 7:35 PM
if there's cgo involved, then we would have to convert everything anyway for portable builds
s

seph

08/18/2022, 7:35 PM
That would be new ground. It feels like an uncomfortable compromise. We’d do it in core, but the dependancies are a hassle. It could be an extension, but there’s value in core.
a

alessandrogario

08/18/2022, 7:35 PM
meaning: cgo would have to go through the osquery-toolchain, otherwise the build won't work on all systems
7:36 PM
If it is pure golang, then I think it should be quite a lot easier
Stefano Bonicatti

Stefano Bonicatti

08/18/2022, 7:36 PM
I’m not sure that’s a good idea (static linking Go libraries to osquery, not even sure it’s possible). If anything I recall that Go doesn’t like to have binaries be stripped
a

alessandrogario

08/18/2022, 7:36 PM
ah interesting, is that for the reflection?
zwass

zwass

08/18/2022, 7:37 PM
Everyone else in this thread is more qualified than me to comment on what the best way to handle the build system issues is, but I do think getting this into core is important for osquery's narrative around containers.
a

alessandrogario

08/18/2022, 7:38 PM
What would be your opinion on shipping core extensions?
7:38 PM
we could even think about having a dedicated startup logic that does not require using --extension or estensions autoload
zwass

zwass

08/18/2022, 7:38 PM
As long as it works out of the box in osquery I think it works. But extensions IME have been really flaky in a thousand different ways.
s

seph

08/18/2022, 7:40 PM
This might not be the right thread, but how have you found them flakey?
zwass

zwass

08/18/2022, 7:41 PM
registration/deregistration issues (often "duplicate blah blah blah" errors), permissions issues, processes not starting/exiting together I've seen all sorts of issues across Launcher, Orbit, and other custom extensions.
s

seph

08/18/2022, 7:42 PM
In the last year, I’ve probably hammered out a bunch of the launcher ones. It’s not perfect, but I found an old, deep race condition.
7:44 PM
If we start from the premise that we want
containerd
tables in osquery, and the only API they provide is grpc, then it feels like an unpleasant choice between:1. pulling in a lot of extensions. And TBH this is hard with our cmake setup 2. Cross language stuff in core. 3. extensions I honestly have no idea what I think. (2) and (3) both unlock some really interesting possibilities. But I think we’d need to think about how to make them feel right.
a

Artemis Tosini

08/18/2022, 7:45 PM
I can fairly easily parse the JSON state files that runc produces but it's a somewhat disjoint set of information from what I can find from the containerd api
a

alessandrogario

08/18/2022, 7:50 PM
It has always seemed like apache thrift is the major culprit, but I am not sure
s

seph

08/18/2022, 7:50 PM
Thrift always feels janky. But launcher had internal race conditions
a

alessandrogario

08/18/2022, 7:51 PM
And I always wondered how much of an effort would it be to essentially reimplement what we have in the extensions folder, and replace Thrift with something else
s

seph

08/18/2022, 7:51 PM
A large ex-employer had a lot of issues with thrift at scale.
7:51 PM
we could use grpc. It’s both awful, and would solve the dependency issue! 😆
zwass

zwass

08/18/2022, 7:52 PM
grpc is by far the standard these days
a

alessandrogario

08/18/2022, 7:52 PM
I don't think it would have to be super fancy, sending JSON through a pipe would not be too hard
7:52 PM
and we would no longer have issues with dealing with libraries which only support one language or the other
zwass

zwass

08/18/2022, 7:52 PM
Yeah, we don't necessarily need any of the features of grpc or thrift
s

seph

08/18/2022, 7:52 PM
Actually, I’m not sure. grpc was new and exciting a bit ago. But I feel like I’m seeing people backing off from it.
a

alessandrogario

08/18/2022, 7:53 PM
we had projects in C++ using it internally, and we are going away from it due to how buggy the C++ sdk is
7:53 PM
and how bad the build system they use is
7:54 PM
But on what seph mentioned: I agree that being able to ship core features in other languages would be nice
s

seph

08/18/2022, 7:54 PM
I think the benefit that things like grpc, thrift, protobufs, etc provide is a structured wire protocol. Clear fields and versioning. That’s hard to do with json
a

alessandrogario

08/18/2022, 7:55 PM
That is true, though I don't remember when the protocol was changed last time
zwass

zwass

08/18/2022, 7:56 PM
Back to the topic at hand... Are folks opposed to bringing containerd tables into core with grpc? We can easily build osquery extensions and bundle them with orbit, but I do feel that it will be a miss for the osquery brand.
a

alessandrogario

08/18/2022, 7:57 PM
it's at least 5 new big dependencies that we have to rewrite in Cmake; I somehow tend to avoid adding new dependencies but I can also see how containerd could be interesting
s

seph

08/18/2022, 7:57 PM
I feel like (1) is okay, but work for people who aren’t me. If y’all are okay with the dependency foot print, I’m okay with it. I think (2) is really interesting and powerful. But I’m not sure how to approach it or what the tradeoffs are. It certainly opens doors.
a

Artemis Tosini

08/18/2022, 7:59 PM
I've dealt with the build system before and can probably figure out (1), though it's a bit more work than I originally bargained for
4:21 PM
I've ported @Stefano Bonicatti’s patch forward to osquery HEAD. It's definitely not in a mergable state but I should be able to clean it up and submit it if all the authors are fine with that (I'd want confirmation so that we don't have copyright uncertainties)https://github.com/artemist-work/osquery/tree/containerd-events
a

alessandrogario

08/30/2022, 5:01 PM
sorry for the late reply; i don't think there's a copyright/license issue (cc @Mike Myers). co-authored-by can be added to include all the original contributors
a

Artemis Tosini

08/30/2022, 5:02 PM
I included that, though there is still the issue of the proto files. I'll work on pulling those from a submodule since I think Apache-2 is incompatible with Apache-2 + GPL