Channels
android_tests
apple-silicon
arm-architecture
auditing-warroom
aws
carving
code-review
community-feeds
core
darkbytes
doorman
ebpf
eclecticiq-polylogyx-extension
extensions
file-carving
fim
fleet
fleet-dev
fleetosquery
foundation
fuzzing
general
golang
goquery
infrastructure
jobs
kolide
linen-dev
linux
macos
officehours
osctrl
plugins
process-auditing
querycon
queryhub
random
selfgroup
sql
tls
uptycs
vendor-feeds
website
windows
zeek
zentral
zercurity
Title
j
jom
08/05/2019, 8:59 AMHi everyone, I’m the guy behind this feature request: https://github.com/osquery/osquery/issues/5678. We will implement this function as soon as possible. For this i have decided to register into this chat 😉. Many greetings Jom
s
seph
08/05/2019, 10:36 AMhi! welcome! have you considered writing it as an extension?
a
alessandrogario
08/05/2019, 10:55 AMHey jom! We are currently changing how dependencies are obtained; not only for technical reasons (better and reproducible builds) but also because we can't easily update the libraries where they are stored now.
If you take seph's approach, then you can more easily bring your own Windows-compatible kakfa library and work around the problem. You can look at the branch I'm working on and copy how I imported it: https://github.com/osquery/osquery/commit/083df997519cf930c8b74f86498b62fb742b07e2
If you decide to work on this, we can help out! 🙂
j
jom
08/05/2019, 11:03 AMThanks for the quick answers! Yes, I see that you are rebuilding quite a bit right now. Just when we finished our poc for the kafka export. 😄 We’ll take a look at the link, thank you very much! Is there already a defined process/pipeline to provide extensions to the the public?
a
alessandrogario
08/05/2019, 11:05 AMExtensions had good support until the huge refactor happened; right now things are still a little messy 😞
If you use osquery 3.3.2 you can more easily build extensions that are easy to deploy
and those extensions are then also compatible with the latest version
We are trying to fix everything and hopefully things will be a lot better by the end of this month
We would like to land new features that add dependencies inside extensions, but for this specific case we would only be enabling what we already have on macOS and Linux (+ some code changes to enable SSL) so I think we can merge this without problem
seph's suggestion is good because you don't have to wait until everything on our side is sorted before you can have the functionality up and running
If you were wondering more about the actual process of copying the extensions (rather than developing them), I think most people are using things like puppet/chef or taking advantage of Kolide products
j
jom
08/05/2019, 11:22 AMmmh. we have other requirements. For us, it’s only a matter of passing on the relevant questions to a kafka. We do nothing on the producing-machine itself. Then a flink job comes to the kafka and processes the data into our PostgreSQL/druid. I am wondering a little bit: The export of data to kafka is integrated into the linux clients. Does it make sense to omit this function from the windows clients?
Only that I just got it right:
- With the extensions plugin in v4.0 there is a lot under construction.
- The same applies to the integrated libraries for 4.0.
it’s not easy to start developing a feature right now, is it?
a
alessandrogario
08/05/2019, 11:32 AMIf it's a new feature (and not enabling what we already have on Linux/macOS) then it's better if it remains inside an extension. I really don't know much about Kafka 😞
The extensions framework is not in 4.0 yet, but it is in 3.3.2. Nothing has changed there. The extension protocol is still compatible, so it's not a problem to build them with 3.3.2 and have them run within 4.0.0 (we will port the SDK soon!)
New features, provided they do not add dependencies, can be implemented right now
I realize the current situation is not ideal, but a lot has happened in the past month (a huge refactor from Facebook + the project has moved under the Linux Foundation)
Both moves have been really good for the health of the project! Things will return to normal once the first stable is out, at the end of this month 🙂
j
jom
08/05/2019, 11:38 AMNo stress. I have just allocated money and resources here and am thinking about how we can get our feature “kafka export for the windows client” most cleverly. Best of all, that it is available for the whole community and we don’t want to start from scratch as soon as 4.0 is out 🙂
a
alessandrogario
08/05/2019, 11:39 AMI think we all agree that we should create a new github.com/osquery/extensions repository
So we can merge it under the osquery organization regardless of whether it's an extension or not
j
jom
08/05/2019, 11:41 AMSorry to ask you again stupidly: In Linux and Mac the kafka exporter works yes. It’s not an extension there, is it?
a
alessandrogario
08/05/2019, 11:42 AMit is currently present inside macOS and Linux, and disabled for Windows
most likely because it was not building
j
jom
08/05/2019, 11:45 AMYes, we wanted to sit on librdkafka. With the intention that it will then build. But that won’t be enough with 4.0, will it?
But in general it makes sense to make kakfa export possible on windows as well, doesn’t it?
Here is it 🙂
https://github.com/osquery/osquery/pull/6095