hi! welcome! have you considered writing it as an extension?
08/05/2019, 10:55 AM
Hey jom! We are currently changing how dependencies are obtained; not only for technical reasons (better and reproducible builds) but also because we can't easily update the libraries where they are stored now.
If you take seph's approach, then you can more easily bring your own Windows-compatible kakfa library and work around the problem. You can look at the branch I'm working on and copy how I imported it: https://github.com/osquery/osquery/commit/083df997519cf930c8b74f86498b62fb742b07e2
If you decide to work on this, we can help out! 🙂
08/05/2019, 11:03 AM
Thanks for the quick answers! Yes, I see that you are rebuilding quite a bit right now. Just when we finished our poc for the kafka export. 😄 We’ll take a look at the link, thank you very much! Is there already a defined process/pipeline to provide extensions to the the public?
08/05/2019, 11:05 AM
Extensions had good support until the huge refactor happened; right now things are still a little messy 😞
If you use osquery 3.3.2 you can more easily build extensions that are easy to deploy
and those extensions are then also compatible with the latest version
We are trying to fix everything and hopefully things will be a lot better by the end of this month
We would like to land new features that add dependencies inside extensions, but for this specific case we would only be enabling what we already have on macOS and Linux (+ some code changes to enable SSL) so I think we can merge this without problem
seph's suggestion is good because you don't have to wait until everything on our side is sorted before you can have the functionality up and running
If you were wondering more about the actual process of copying the extensions (rather than developing them), I think most people are using things like puppet/chef or taking advantage of Kolide products
08/05/2019, 11:22 AM
mmh. we have other requirements. For us, it’s only a matter of passing on the relevant questions to a kafka. We do nothing on the producing-machine itself. Then a flink job comes to the kafka and processes the data into our PostgreSQL/druid. I am wondering a little bit: The export of data to kafka is integrated into the linux clients. Does it make sense to omit this function from the windows clients?
Only that I just got it right:
- With the extensions plugin in v4.0 there is a lot under construction.
- The same applies to the integrated libraries for 4.0.
it’s not easy to start developing a feature right now, is it?
08/05/2019, 11:32 AM
If it's a new feature (and not enabling what we already have on Linux/macOS) then it's better if it remains inside an extension. I really don't know much about Kafka 😞
The extensions framework is not in 4.0 yet, but it is in 3.3.2. Nothing has changed there. The extension protocol is still compatible, so it's not a problem to build them with 3.3.2 and have them run within 4.0.0 (we will port the SDK soon!)
New features, provided they do not add dependencies, can be implemented right now
I realize the current situation is not ideal, but a lot has happened in the past month (a huge refactor from Facebook + the project has moved under the Linux Foundation)
Both moves have been really good for the health of the project! Things will return to normal once the first stable is out, at the end of this month 🙂
08/05/2019, 11:38 AM
No stress. I have just allocated money and resources here and am thinking about how we can get our feature “kafka export for the windows client” most cleverly. Best of all, that it is available for the whole community and we don’t want to start from scratch as soon as 4.0 is out 🙂