https://github.com/osquery/osquery logo
Title
a

Artemis Tosini

08/18/2022, 7:13 PM
I'm looking at adding some containerd tables. Unfortunately the API is via grpc so we'd require grpc as a dependancy and either pregenerate the C++ with protoc or use protoc at build time, does anyone have opinions?
z

zwass

08/18/2022, 7:19 PM
I think it's reasonable to use the grpc API (it's connecting only over a unix domain socket, not a true remote API). Between pregenerate and build time, I don't have much preference... Whatever is cleaner/easier sgtm.
cc @alessandrogario @Stefano Bonicatti
a

alessandrogario

08/18/2022, 7:21 PM
I seem to recall what was the hiccup, I think the grpc library is rather heavy and it comes with several additional libraries
They would all need to be converted to CMake
Going by memory: at least absl, cares, protobuf and probably a couple of additional ones
They have a 22k lines CMakeLists.txt file
We have an impl of that in our https://github.com/trailofbits/osquery fork, though it's from 2020 and was no longer updated
I think a PR was never made because the amount of code and dependencies it added was quite big
z

zwass

08/18/2022, 7:28 PM
Are you opposed to adding it? Or just warning that it's nontrivial?
a

alessandrogario

08/18/2022, 7:30 PM
I think it's going to generate some maintenance work if we want to do it properly
compared to C++, golang could implement this more easily
I think the major fault here is the grpc C++ SDK which is quite messy and even lacks features (async was only half-implemented)
s

seph

08/18/2022, 7:32 PM
I suspect that the go side still pulls in a lot of libs, but the build system hides that mess from you.
a

Artemis Tosini

08/18/2022, 7:33 PM
Yeah, that's amost certainly true. It's probably still pulling in protoc
z

zwass

08/18/2022, 7:33 PM
We could definitely build these tables as a Go extension, but I suspect folks might be disappointed it doesn't go into core osquery?
Or, how crazy would it be to try to build a library in Go and then static link and call that from osquery?
a

Artemis Tosini

08/18/2022, 7:34 PM
I'm not sure how much Go would like that, it needs to run its gc
z

zwass

08/18/2022, 7:34 PM
That would open the floodgates to tons of other tables being built in Go, which would certainly make building osquery core tables much more accessible.
a

alessandrogario

08/18/2022, 7:34 PM
I think it depends on how much cgo is present in there
z

zwass

08/18/2022, 7:34 PM
Cursory searching indicates Go can be called from C++ but I have no idea how that works with GC.
a

alessandrogario

08/18/2022, 7:35 PM
if there's cgo involved, then we would have to convert everything anyway for portable builds
s

seph

08/18/2022, 7:35 PM
That would be new ground. It feels like an uncomfortable compromise. We’d do it in core, but the dependancies are a hassle. It could be an extension, but there’s value in core.
a

alessandrogario

08/18/2022, 7:35 PM
meaning: cgo would have to go through the osquery-toolchain, otherwise the build won't work on all systems
If it is pure golang, then I think it should be quite a lot easier
s

Stefano Bonicatti

08/18/2022, 7:36 PM
I’m not sure that’s a good idea (static linking Go libraries to osquery, not even sure it’s possible). If anything I recall that Go doesn’t like to have binaries be stripped
a

alessandrogario

08/18/2022, 7:36 PM
ah interesting, is that for the reflection?
z

zwass

08/18/2022, 7:37 PM
Everyone else in this thread is more qualified than me to comment on what the best way to handle the build system issues is, but I do think getting this into core is important for osquery's narrative around containers.
a

alessandrogario

08/18/2022, 7:38 PM
What would be your opinion on shipping core extensions?
we could even think about having a dedicated startup logic that does not require using --extension or estensions autoload
z

zwass

08/18/2022, 7:38 PM
As long as it works out of the box in osquery I think it works. But extensions IME have been really flaky in a thousand different ways.
s

seph

08/18/2022, 7:40 PM
This might not be the right thread, but how have you found them flakey?
z

zwass

08/18/2022, 7:41 PM
registration/deregistration issues (often "duplicate blah blah blah" errors), permissions issues, processes not starting/exiting together I've seen all sorts of issues across Launcher, Orbit, and other custom extensions.
s

seph

08/18/2022, 7:42 PM
In the last year, I’ve probably hammered out a bunch of the launcher ones. It’s not perfect, but I found an old, deep race condition.
If we start from the premise that we want
containerd
tables in osquery, and the only API they provide is grpc, then it feels like an unpleasant choice between: 1. pulling in a lot of extensions. And TBH this is hard with our cmake setup 2. Cross language stuff in core. 3. extensions I honestly have no idea what I think. (2) and (3) both unlock some really interesting possibilities. But I think we’d need to think about how to make them feel right.
a

Artemis Tosini

08/18/2022, 7:45 PM
I can fairly easily parse the JSON state files that runc produces but it's a somewhat disjoint set of information from what I can find from the containerd api
a

alessandrogario

08/18/2022, 7:50 PM
It has always seemed like apache thrift is the major culprit, but I am not sure
s

seph

08/18/2022, 7:50 PM
Thrift always feels janky. But launcher had internal race conditions
a

alessandrogario

08/18/2022, 7:51 PM
And I always wondered how much of an effort would it be to essentially reimplement what we have in the extensions folder, and replace Thrift with something else
s

seph

08/18/2022, 7:51 PM
A large ex-employer had a lot of issues with thrift at scale.
we could use grpc. It’s both awful, and would solve the dependency issue! 😆
z

zwass

08/18/2022, 7:52 PM
grpc is by far the standard these days
a

alessandrogario

08/18/2022, 7:52 PM
I don't think it would have to be super fancy, sending JSON through a pipe would not be too hard
and we would no longer have issues with dealing with libraries which only support one language or the other
z

zwass

08/18/2022, 7:52 PM
Yeah, we don't necessarily need any of the features of grpc or thrift
s

seph

08/18/2022, 7:52 PM
Actually, I’m not sure. grpc was new and exciting a bit ago. But I feel like I’m seeing people backing off from it.
a

alessandrogario

08/18/2022, 7:53 PM
we had projects in C++ using it internally, and we are going away from it due to how buggy the C++ sdk is
and how bad the build system they use is
But on what seph mentioned: I agree that being able to ship core features in other languages would be nice
s

seph

08/18/2022, 7:54 PM
I think the benefit that things like grpc, thrift, protobufs, etc provide is a structured wire protocol. Clear fields and versioning. That’s hard to do with json
a

alessandrogario

08/18/2022, 7:55 PM
That is true, though I don't remember when the protocol was changed last time
z

zwass

08/18/2022, 7:56 PM
Back to the topic at hand... Are folks opposed to bringing containerd tables into core with grpc? We can easily build osquery extensions and bundle them with orbit, but I do feel that it will be a miss for the osquery brand.
a

alessandrogario

08/18/2022, 7:57 PM
it's at least 5 new big dependencies that we have to rewrite in Cmake; I somehow tend to avoid adding new dependencies but I can also see how containerd could be interesting
s

seph

08/18/2022, 7:57 PM
I feel like (1) is okay, but work for people who aren’t me. If y’all are okay with the dependency foot print, I’m okay with it. I think (2) is really interesting and powerful. But I’m not sure how to approach it or what the tradeoffs are. It certainly opens doors.
a

Artemis Tosini

08/18/2022, 7:59 PM
I've dealt with the build system before and can probably figure out (1), though it's a bit more work than I originally bargained for
I've ported @Stefano Bonicatti’s patch forward to osquery HEAD. It's definitely not in a mergable state but I should be able to clean it up and submit it if all the authors are fine with that (I'd want confirmation so that we don't have copyright uncertainties) https://github.com/artemist-work/osquery/tree/containerd-events
a

alessandrogario

08/30/2022, 5:01 PM
sorry for the late reply; i don't think there's a copyright/license issue (cc @Mike Myers). co-authored-by can be added to include all the original contributors
a

Artemis Tosini

08/30/2022, 5:02 PM
I included that, though there is still the issue of the proto files. I'll work on pulling those from a submodule since I think Apache-2 is incompatible with Apache-2 + GPL