Title
#windows
y

Yves Dolce

01/27/2020, 9:44 PM
Can a Windows process run OSQUERY queries using (I guess) the ThriftAPI? If so, is it documented? Thanks. (BTW, the Thrift API https://github.com/osquery/osquery/blob/master/osquery.thrift link on https://osquery.readthedocs.io/en/stable/development/osquery-sdk/#thrift-api takes you to a github 404)
a

alessandrogario

01/28/2020, 12:30 AM
you can probably use selectAllFrom (look in the osquery source code). You can also call the registry directly, see here: https://github.com/zeek/zeek-agent/blob/master/components/zeekosqueryinterface/src/osquerytableplugin.cpp#L43
theopolis

theopolis

01/28/2020, 1:20 AM
Do you mind creating a GitHub issue for the broken link?
y

Yves Dolce

01/28/2020, 6:51 PM
Thanks @alessandrogario. I have a hard time finding such an example that would target Windows. Do you know any?
a

alessandrogario

01/28/2020, 7:13 PM
There are no Windows specifics, just link the source folder under external and it should work
7:13 PM
you can follow the build guide in the readme to setup the environment
y

Yves Dolce

01/28/2020, 7:25 PM
I was hoping for something that would not have involved the osquery tree (symlink'ing is still bringing the code in the tree). One that could be built with a PS1/bat or a
vcxproj
, or even a
CMakeFile
, but completely outside osquery. One that would reference its own boost/rapidjson/googletest/sqllite/gflags/... Am I making sense?
a

alessandrogario

01/28/2020, 7:26 PM
it is not supported, so it's uncharted territory
y

Yves Dolce

01/28/2020, 7:26 PM
Understood but are you aware of anyone that has done so w/ osquery 4.x?
a

alessandrogario

01/28/2020, 7:27 PM
no, as far as I know
7:28 PM
refactoring the SDK to make it standalone is in wishlist but not currently planned
y

Yves Dolce

01/28/2020, 7:28 PM
Thanks for your patience. Buena serata!
theopolis

theopolis

01/28/2020, 10:07 PM
I might not have all the context for what you are trying to do @Yves Dolce, but there are may ways a process on Windows can execute queries using the Thrift API within modifying osquery source code. What language is your Windows process/program written in? C/C++?
y

Yves Dolce

01/28/2020, 10:07 PM
Yes.
10:11 PM
To be honest with you, I would have expected an SDK that included headers and libraries and possibly a couple of tools required to build ones executable. Not the current documented way.
a

alessandrogario

01/28/2020, 10:12 PM
There are many issues that the current implementation solves that may not be apparent from the outside
10:12 PM
portability is the most important one, as it's extremely hard to build once and deploy everywhere when using Linux
10:13 PM
and that not only has to do with what you build but also what you link when generating a binary
10:13 PM
i.e.: you can't just link against any boost library and expect it to work on other distributions that are different than the one used to build
y

Yves Dolce

01/28/2020, 10:13 PM
Agreed. But you can build the BOOST libs for windows and thus do it, can't you?
a

alessandrogario

01/28/2020, 10:14 PM
the current way solves this and allow you to just focus on the extension code rather than forcing to think about how to build a toolchain that can do that, and then rebuilt all your dependencies with it
10:15 PM
you can't really, as you are linking against osquery libraries
10:15 PM
that are already importing boost
10:15 PM
you have to use the same boost used to build osquery (as long as you are linking against osquery libs)
10:15 PM
and that is true not just for boost but for any other library
10:15 PM
CRT too, you don't want to mix different CRT libraries in the same process
y

Yves Dolce

01/28/2020, 10:15 PM
THat was just an example of how an open source project (i.e. BOOST) solve those issues.
a

alessandrogario

01/28/2020, 10:16 PM
Boost doesn't solve the issue, and actually make it worse
10:16 PM
they are using their own build system called B2 that does not have good support for external toolchains
y

Yves Dolce

01/28/2020, 10:16 PM
Those are the positive sides to the current solution. But the negative side is that I can't build an extension if I don't have the whole osquery enchilada built at the same time on my build machine.
a

alessandrogario

01/28/2020, 10:16 PM
and we basically had to rewrite it in CMake
y

Yves Dolce

01/28/2020, 10:17 PM
What I'm saying is that somehow, you can get os and version specific boost libs and use them when building your project.
a

alessandrogario

01/28/2020, 10:18 PM
but that will not work and potentially cause issue
10:18 PM
when linking the osquery SDK, the binary should link the same exact boost used to build osquery
10:18 PM
and that can't be downloaded on the boost website
10:19 PM
it doesn't just have to be the same exact version, it has to be same lib files
10:19 PM
because compilation flags, settings, definition, compiler used, CRT used will greatly vary the lib files
10:19 PM
even if the boost version is the same
10:21 PM
the code written in your extension (compiled with a certain version of Boost/CRT/etc) can pass objects to osquery code (that may have been compiled with a different Boost)
10:24 PM
It is possible to write a brand new SDK that is entirely based on just the thrift protocol specification (like the Go extension SDK). This would have 0 dependencies on osquery and could work with any library of your choice.
y

Yves Dolce

01/28/2020, 10:24 PM
Yes. But those issue are not specific to osquery. Windows has it and provide C/COM SDKs, all the OSes have that issue.
a

alessandrogario

01/28/2020, 10:25 PM
Going down this path sounds cool but unlike Go it will be impossible to redistribute binaries without a proper setup
10:25 PM
most projects will attempt to implement reproducible builds, and using system libraries (i.e. Boost from the Ubuntu repository) or downloaded ones (from the Boost website) is never going to work
y

Yves Dolce

01/28/2020, 10:26 PM
I don't even want to try that 🙂 I'd rather listen to your wise advices and do the supported thing!
a

alessandrogario

01/28/2020, 10:27 PM
We currently do not have that issue, since we have reproducible (save for a small chunk related to glibc, sadly) builds
10:27 PM
(and you can check this with ldd -d osqueryd)
10:27 PM
or on WIndows, you can use something like the CFF Explorer
10:27 PM
to look at the import table
10:27 PM
(or Dependency Walker)
10:28 PM
But I feel your pain, I'd like to be able to build with a standalone SDK
10:28 PM
a standalone SDK could also be shipped with the osquery package
10:29 PM
we want that (separate SDK that can be built with the custom toolchain so we can redistribute binaries), but unless someone sponsors that work it's hard to prioritize it
10:30 PM
since it's not really causing major issues now compared to other high value targets (like better container introspection)
y

Yves Dolce

01/28/2020, 10:30 PM
And I understand the osquery project point of view.
10:32 PM
This was a very useful conversation. Thank you so much @alessandrogario.
a

alessandrogario

01/28/2020, 10:32 PM
Sample use case: we have one of the Arch Linux maintainers here (Anatol) that really needs to use system libraries to package osquery for the distribution repositories
10:32 PM
if you are familar with Thrift maybe we can get something started
10:32 PM
if it's a strong requirement for you
y

Yves Dolce

01/28/2020, 10:33 PM
I'd love to but I'd first need to have a closer look at Thrift API.
1:14 AM
I just did and experimented with
osquery\extensions\thrift\osquery.thrift
. Again, I'm trying to perform osquery queries and get the corresponding result. That can be done using only Thrift?
theopolis

theopolis

01/29/2020, 2:48 AM
It gets you 50% of the way, take a look at the go implementation for an example https://github.com/kolide/osquery-go/blob/master/README.md one could create a new repo called osquery-cpp and write code similar to what you see in that project
y

Yves Dolce

01/29/2020, 6:31 PM
Thanks but is this not also "not supported and uncharted" as @alessandrogario mentioned earlier here when I was entertaining the idea of building an extension/plugin "outside the osquery tree"?