Title
#general
Stefano Bonicatti

Stefano Bonicatti

02/15/2020, 12:40 PM
@Prateek Kumar Nischal the master branch of our repo is still targeting osquery 3.3.2, for osquery 4.x you have to check this branch https://github.com/trailofbits/osquery-extensions/tree/extension_porting_4.0.1
p

Prateek Kumar Nischal

02/15/2020, 12:58 PM
Thanks @Stefano Bonicatti, I did move to
$ git br
* extension_porting_4.0.1 6c0f9b3 [origin/extension_porting_4.0.1] Other cleanups
  master                  9a6be96 [origin/master] Improve build instructions (#49)
PS: I have removed all extensions for now.. I am just using the
darwin_unified_log
. Also, modified the submodule config to only include json11.
1:00 PM
It's linked and still I get the same issue.. (running from the osquery project root)
$ make externals
Built target externals
Exits at this
Stefano Bonicatti

Stefano Bonicatti

02/15/2020, 1:02 PM
How does the CMake configure log looks like?
p

Prateek Kumar Nischal

02/15/2020, 1:04 PM
CMake run at osquery build dir.. ?
Stefano Bonicatti

Stefano Bonicatti

02/15/2020, 1:06 PM
I mean when you run
cmake -DOSQUERY_TOOLCHAIN_SYSROOT=...
from the build folder, what was the whole output
p

Prateek Kumar Nischal

02/15/2020, 1:07 PM
I was following:
# Download source
git clone <https://github.com/osquery/osquery>
cd osquery

# Configure
mkdir build; cd build
cmake -DCMAKE_OSX_DEPLOYMENT_TARGET=10.11 ..

# Build
cmake --build .
and this runs fine..
1:08 PM
$ cmake -DCMAKE_OSX_DEPLOYMENT_TARGET=10.11 ..
CMake Warning (dev) in CMakeLists.txt:
  No project() command is present.  The top-level CMakeLists.txt file must
  contain a literal, direct call to the project() command.  Add a line of
  code such as

    project(ProjectName)

  near the top of the file, but after cmake_minimum_required().

  CMake is pretending there is a "project(Project)" command on the first
  line.
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) in CMakeLists.txt:
  No cmake_minimum_required command is present.  A line of code such as

    cmake_minimum_required(VERSION 3.16)

  should be added at the top of the file.  The version specified may be lower
  if you wish to support older CMake versions for this project.  For more
  information run "cmake --help-policy CMP0000".
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Configuring done
-- Generating done
-- Build files have been written to: /Users/p0n002h/git/osquery
Stefano Bonicatti

Stefano Bonicatti

02/15/2020, 1:11 PM
So I do see that the
externals
target indeed doesn't trigger the build, I'll try to understand why, though those messages you see during configuration are not a good sign. How exactly did you remove the other extensions etc?
p

Prateek Kumar Nischal

02/15/2020, 1:13 PM
Just rm.. and then edit the submodule files.. I saw the CMake file.. it didn't have any explicit mention of the specific folders.. so I did not touch that
1:20 PM
looks like my osquery build broke.. now
cmake --build .
is throwing missing openssl cmake file errors..
make[2]: *** No rule to make target `installed_formulas/openssl/lib/libssl.a', needed by `osquery/osqueryd'.  Stop.
make[1]: *** [osquery/CMakeFiles/osqueryd.dir/all] Error 2
make: *** [all] Error 2
1:20 PM
i'll try to start from scratch
Stefano Bonicatti

Stefano Bonicatti

02/15/2020, 1:31 PM
There's a small update I have to do on our extensions due to the last release, which is not the reason of this failure, but that it will make them fail to compile later. Also, if you want to specify which extensions to build, please use the env var
TRAILOFBITS_EXTENSIONS_TO_BUILD
; this is mentioned in the README too.
1:33 PM
Also, the
externals
target right now only triggers the build of extensions that do not have their own CMakeLists.txt
1:34 PM
In this case you have to build the
trailofbits_osquery_extensions.ext
target. You can find that by running
make help | grep trailofbits
p

Prateek Kumar Nischal

02/15/2020, 1:39 PM
I had read about the
TRAILOFBITS_EXTENSIONS_TO_BUILD
, the reason of deleting the other extensions was to ignore the submodule clone, it was taking wayyy to long to clone the Pcap++.
Stefano Bonicatti

Stefano Bonicatti

02/15/2020, 1:46 PM
I'm not sure I follow, it should be the
git clone --recurse-submodules <url>
which is cloning all submodules, it's not necessary to delete anything, you can just drop the option from the clone and then
git submodule update --init <submodule path>
on what you need.
p

Prateek Kumar Nischal

02/15/2020, 1:48 PM
yes.. that's what I realised later.. 😅 That;s just my bad..
Stefano Bonicatti

Stefano Bonicatti

02/15/2020, 2:07 PM
Don't worry! There are still a couple of rough edges.
p

Prateek Kumar Nischal

02/15/2020, 2:08 PM
after running the configure, I was not able to find the trailofbits.. 😕. The configure and build ran successfully..
Stefano Bonicatti

Stefano Bonicatti

02/15/2020, 2:11 PM
That might be caused by a typo in the extensions project name I just fixed ahah
2:11 PM
might want to pull again
2:12 PM
actually no, it shouldn't be that; you mean you tried to
make help | grep trailofbits
and it was empty?
2:13 PM
did the extensions build at all?
p

Prateek Kumar Nischal

02/15/2020, 2:14 PM
I had started from scratch.. so I just ran the configure and
cmake --build .
for now and the trailofbits project is linked in the
externals
directory
2:14 PM
do I need to run anything else.. ?
Stefano Bonicatti

Stefano Bonicatti

02/15/2020, 2:17 PM
if you do see
<build folder>/external/<link name>/trailofbits_osquery_extensions.ext
It mean the extensions are built. I was wondering though why you said that you weren't able to find
trailofbits
. You mean the target or the extension binary?
p

Prateek Kumar Nischal

02/15/2020, 2:17 PM
$ pwd
/Users/p0n002h/git/osquery/build/external
$ ls
CMakeFiles          Makefile            cmake_install.cmake
2:18 PM
I was not able to find the target
trailofbits
2:18 PM
It's empty.. so the extension did not build
2:20 PM
Hey.. by the way.. appreciate it man.. thanks for helping me.. 😄
2:23 PM
The funny thing is that I can't even find the
example
extension file too.. 😕
$ pwd
/Users/p0n002h/git/osquery
$ find . -name "example_extension.ext"
$
3:43 PM
@Stefano Bonicatti Any luck.. ?
Stefano Bonicatti

Stefano Bonicatti

02/15/2020, 3:45 PM
could you delete the
CMakeCache.txt
file in the osquery build folder, then run the cmake configure again and copy here the log?
p

Prateek Kumar Nischal

02/15/2020, 4:33 PM
Looks more interesting this time.
-- ================================
-- Trail of Bits osquery extensions
-- Version: 1.2
-- ================================
Importing libraries from '/Users/p0n002h/git/osquery/external/extension_trailofbits/libraries/'
 > extutils
 > firewall
 > hostsfile
Importing extensions from '/Users/p0n002h/git/osquery/external/extension_trailofbits'
 ! Missing '.platforms` file: CMakeFiles. Skipping...
 > darwin_unified_log
 > efigy
-- Found CURL: /usr/lib/libcurl.dylib (found version "7.54.0")
 > fwctl
 > santa
-- ================================
4:35 PM
It did pick the files 😄
4:38 PM
It failed the build though.. looks like some autogen code.
4:39 PM
/Users/p0n002h/git/osquery/build/external/extension_trailofbits/osquery_extension_group_main.cpp:42:10: error: no member named 'waitThenShutdown' in 'osquery::Initializer'; did you mean 'waitForShutdown'?
  runner.waitThenShutdown();
         ^~~~~~~~~~~~~~~~
         waitForShutdown
Stefano Bonicatti

Stefano Bonicatti

02/15/2020, 4:43 PM
yep that is what I was talking about earlier, you do need to change to that function. For now you can do so in the
tools/codegen/templates/osquery_extension_group_main.cpp.in
file
p

Prateek Kumar Nischal

02/15/2020, 4:51 PM
Yes.. I did change the file and I am able to build
Stefano Bonicatti

Stefano Bonicatti

02/15/2020, 4:53 PM
good!
4:53 PM
meanwhile I've opened a PR to fix that
4:54 PM
Ah I see you just opened one, keep yours 🙂
p

Prateek Kumar Nischal

02/15/2020, 4:55 PM
😅
4:55 PM
thanks..