I m looking at adding some containerd tables Unfortunately t

Question

I m looking at adding some containerd tables Unfortunately the API is via grpc so we d require grpc as a dependancy and either pregenerate the C++ with protoc or use protoc at build time does anyone have opinions

zwass · Answer

I think it s reasonable to use the grpc API it s connecting only over a unix domain socket not a true remote API Between pregenerate and build time I don t have much preference Whatever is cleaner easier sgtm

zwass · Answer

cc < alessandrogario> < Stefano Bonicatti>

alessandrogario · Answer

I seem to recall what was the hiccup I think the grpc library is rather heavy and it comes with several additional libraries

alessandrogario · Answer

They would all need to be converted to CMake

alessandrogario · Answer

Going by memory at least absl cares protobuf and probably a couple of additional ones

alessandrogario · Answer

The grpc CMake is also rather messy <https github com grpc grpc blob master CMakeLists txt L21985>

alessandrogario · Answer

They have a 22k lines CMakeLists txt file

alessandrogario · Answer

We have an impl of that in our <https github com trailofbits osquery> fork though it s from 2020 and was no longer updated

alessandrogario · Answer

I think a PR was never made because the amount of code and dependencies it added was quite big

zwass · Answer

Are you opposed to adding it Or just warning that it s nontrivial

Stefano Bonicatti · Answer

alessandrogario · Answer

I think it s going to generate some maintenance work if we want to do it properly

alessandrogario · Answer

compared to C++ golang could implement this more easily

alessandrogario · Answer

I think the major fault here is the grpc C++ SDK which is quite messy and even lacks features async was only half implemented

seph · Answer

I suspect that the go side still pulls in a lot of libs but the build system hides that mess from you

Artemis Tosini · Answer

Yeah that s amost certainly true It s probably still pulling in protoc

zwass · Answer

We could definitely build these tables as a Go extension but I suspect folks might be disappointed it doesn t go into core osquery

zwass · Answer

Or how crazy would it be to try to build a library in Go and then static link and call that from osquery

Artemis Tosini · Answer

I m not sure how much Go would like that it needs to run its gc

zwass · Answer

That would open the floodgates to tons of other tables being built in Go which would certainly make building osquery core tables much more accessible

alessandrogario · Answer

I think it depends on how much cgo is present in there

zwass · Answer

Cursory searching indicates Go can be called from C++ but I have no idea how that works with GC

alessandrogario · Answer

if there s cgo involved then we would have to convert everything anyway for portable builds

seph · Answer

That would be new ground It feels like an uncomfortable compromise We d do it in core but the dependancies are a hassle It could be an extension but there s value in core

alessandrogario · Answer

meaning cgo would have to go through the osquery toolchain otherwise the build won t work on all systems

alessandrogario · Answer

If it is pure golang then I think it should be quite a lot easier

Stefano Bonicatti · Answer

I m not sure that s a good idea static linking Go libraries to osquery not even sure it s possible If anything I recall that Go doesn t like to have binaries be stripped

alessandrogario · Answer

ah interesting is that for the reflection

zwass · Answer

Everyone else in this thread is more qualified than me to comment on what the best way to handle the build system issues is but I do think getting this into core is important for osquery s narrative around containers

alessandrogario · Answer

What would be your opinion on shipping core extensions

alessandrogario · Answer

we could even think about having a dedicated startup logic that does not require using extension or estensions autoload

zwass · Answer

As long as it works out of the box in osquery I think it works But extensions IME have been really flaky in a thousand different ways

seph · Answer

This might not be the right thread but how have you found them flakey

zwass · Answer

registration deregistration issues often duplicate blah blah blah errors permissions issues processes not starting exiting together I ve seen all sorts of issues across Launcher Orbit and other custom extensions

seph · Answer

In the last year I ve probably hammered out a bunch of the launcher ones It s not perfect but I found an old deep race condition

seph · Answer

If we start from the premise that we want `containerd` tables in osquery and the only API they provide is grpc then it feels like an unpleasant choice between 1 pulling in a lot of extensions And TBH this is hard with our cmake setup 2 Cross language stuff in core 3 extensions I honestly have no idea what I think 2 and 3 both unlock some really interesting possibilities But I think we d need to think about how to make them feel right

Artemis Tosini · Answer

I can fairly easily parse the JSON state files that runc produces but it s a somewhat disjoint set of information from what I can find from the containerd api

alessandrogario · Answer

It has always seemed like apache thrift is the major culprit but I am not sure

seph · Answer

Thrift always feels janky But launcher had internal race conditions

alessandrogario · Answer

And I always wondered how much of an effort would it be to essentially reimplement what we have in the extensions folder and replace Thrift with something else

seph · Answer

A large ex employer had a lot of issues with thrift at scale

seph · Answer

we could use grpc It s both awful and would solve the dependency issue laughing

zwass · Answer

grpc is by far the standard these days

alessandrogario · Answer

I don t think it would have to be super fancy sending JSON through a pipe would not be too hard

alessandrogario · Answer

and we would no longer have issues with dealing with libraries which only support one language or the other

zwass · Answer

Yeah we don t necessarily need any of the features of grpc or thrift

seph · Answer

Actually I m not sure grpc was new and exciting a bit ago But I feel like I m seeing people backing off from it

alessandrogario · Answer

we had projects in C++ using it internally and we are going away from it due to how buggy the C++ sdk is

alessandrogario · Answer

and how bad the build system they use is

alessandrogario · Answer

But on what seph mentioned I agree that being able to ship core features in other languages would be nice

seph · Answer

I think the benefit that things like grpc thrift protobufs etc provide is a structured wire protocol Clear fields and versioning That s hard to do with json

alessandrogario · Answer

That is true though I don t remember when the protocol was changed last time

zwass · Answer

Back to the topic at hand Are folks opposed to bringing containerd tables into core with grpc We can easily build osquery extensions and bundle them with orbit but I do feel that it will be a miss for the osquery brand

seph · Answer

I ll re iterate <https osquery slack com archives CBLGAN1HD p1660851855917469 thread ts=1660850035 080819 amp cid=CBLGAN1HD>

alessandrogario · Answer

it s at least 5 new big dependencies that we have to rewrite in Cmake I somehow tend to avoid adding new dependencies but I can also see how containerd could be interesting

seph · Answer

I feel like 1 is okay but work for people who aren t me If y all are okay with the dependency foot print I m okay with it I think 2 is really interesting and powerful But I m not sure how to approach it or what the tradeoffs are It certainly opens doors

Artemis Tosini · Answer

I ve dealt with the build system before and can probably figure out 1 though it s a bit more work than I originally bargained for

Artemis Tosini · Answer

I ve ported < Stefano Bonicatti> s patch forward to osquery HEAD It s definitely not in a mergable state but I should be able to clean it up and submit it if all the authors are fine with that I d want confirmation so that we don t have copyright uncertainties

alessandrogario · Answer

sorry for the late reply i don t think there s a copyright license issue cc < Mike Myers> co authored by can be added to include all the original contributors

Artemis Tosini · Answer

I included that though there is still the issue of the proto files I ll work on pulling those from a submodule since I think Apache 2 is incompatible with Apache 2 + GPL