Title
#general
a

Ahmed

11/09/2020, 1:20 PM
Hey Folks, trying to build network_monitor extension from trailofbits followed the build guide for osquery and their github page. keep getting fatel errors. i’m building in a fresh debian:latest docker image. any advise, i’m only interested in building the extensions and not osquery itself. thanks a lot.
Mike Myers

Mike Myers

11/09/2020, 8:58 PM
The extensions do require all of osquery to be built along with them, so they can't be compiled separately
a

Ahmed

11/10/2020, 9:10 AM
ok, that what i followed actually the compilation steps in osquery docs and their github which gave me those errors and keeps failing. any advise/help how to get a clean compilation. thanks a lot
3:34 PM
@Mike Myers getting same failure on different debian versions and ubuntu versions. and this is the command i’m following
apt-get update -y
apt-get install git curl cmake python3 git python3 bison flex make wget -y
pip3 install timeout_decorator thrift==0.11.0 osquery pexpect==3.3
apt install --no-install-recommends rpm binutils -y
#wget <https://github.com/osquery/osquery-toolchain/releases/download/1.1.0/osquery-toolchain-1.1.0-x86_64.tar.xz>
tar xvf osquery-toolchain-1.1.0-x86_64.tar.xz -C /usr/local

git clone <https://github.com/osquery/osquery>

git clone --recurse-submodules <https://github.com/trailofbits/osquery-extensions.git>

#wget <https://github.com/Kitware/CMake/releases/download/v3.14.6/cmake-3.14.6-Linux-x86_64.tar.gz>
tar xvf cmake-3.14.6-Linux-x86_64.tar.gz -C /usr/local --strip 1
export PATH=/usr/local/bin:$PATH

cd osquery
ln -s ../../osquery-extensions ./external/extension_trailofbits

mkdir build; cd build
cmake -DOSQUERY_TOOLCHAIN_SYSROOT=/usr/local/osquery-toolchain ..
cmake --build . -j30 # where 30 is the number of parallel build jobs
Mike Myers

Mike Myers

11/10/2020, 4:39 PM
Yea those are weird errors, but when LLVM fails during compilation sometimes it's because your build system doesn't have enough RAM
4:40 PM
@Stefano Bonicatti what's the recommended RAM for build systems? Like 6GB?
Stefano Bonicatti

Stefano Bonicatti

11/10/2020, 4:42 PM
Depends on how many jobs you run. ~4GB are required to build one of the C++ files from the AWS SDK and that's the only one really that has that huge requirement. 6GB might be a safe bet for maybe up to 4 jobs.
4:45 PM
Sometimes I run with 8GBs on my Linux VM (Ubuntu) and that's fine for 10 jobs.. but not always, also depends on what else it was doing. For a 30 jobs build I would give it at least 16GBs.
4:50 PM
But again those are somewhat estimates.. the growth is non linear, and it highly depends on what each job is doing and how many can run really parallel.. so a bit of testing is required ^^'
Mike Myers

Mike Myers

11/10/2020, 4:52 PM
yea using
-j30
could be increasing your memory overhead too much and one of the parallelized jobs is failing
a

Ahmed

11/10/2020, 4:57 PM
it was failing before with
-j10
and i have enough ram, but i will take your advise and test with much lower number. thanks a lot both of you
Stefano Bonicatti

Stefano Bonicatti

11/10/2020, 4:58 PM
@Ahmed If the error is that clang gets killed it's most likely due to memory issues. How much memory do you have?
Mike Myers

Mike Myers

11/10/2020, 5:19 PM
Yea the OpenSSL library in particular is prone to failing in low memory situations, in my experience
a

Ahmed

11/11/2020, 8:21 AM
i have 16G on my mac which i’m running the docker on. and now i ran the same build with
-j4
but still getting killed.
Stefano Bonicatti

Stefano Bonicatti

11/11/2020, 10:02 AM
@Ahmed could you please check via
free -m
inside the Docker image that you have all 16GB? Here https://docs.docker.com/docker-for-mac/#:~:text=Memory%3A%20By%20default%2C%20Docker%20Desktop,swap%20file%20size%20as%20needed. it seems like the default is to use only 2GB from the Host.
a

Ahmed

11/11/2020, 10:03 AM
i just ran it on another system with
-j4
and it did finish compiling. but when i tried to run the
osqueryi --extension osquery/build/external/trailofbits_osquery_extensions.ext
i dont see the tables mentioned, so trying to figure out why 😄
10:15 AM
i only see these tables i guess:
iptables_ext
ip6tables_ext
and in side the build i see these files. i’m only interested to get the extension for dns. thanks a lot for taking the time to look at this.
12:17 PM
the
dns_events
doesnot work when i run the extension using this way
osqueryi --allow_unsafe  --extension osquery/build/external/extension_trailofbits/trailofbits_osquery_extensions.ext
and i get this message when quitting osquery
Thrift: Wed Nov 11 07:15:12 2020 TSocket::open() connect() <Host:  Port: 0>No such file or directory
but it works with this way and generate info/error message and doenot get killed when i quit osquery i have to explicitly run
kill
command against it.
osqueryi --allow_unsafe  --extension osquery/build/external/extension_trailofbits/extensions/network_monitor/network_monitor.ext
and it keep generating this
I1111 07:16:53.047338 27998 extensions.cpp:298] Extension watcher ending: osquery core has gone away
10:16 AM
@Stefano Bonicatti any advise why the extension doesnot terminate after quitting osquery and keeps running and outputing to the screen.
I1111 07:16:53.047338 27998 extensions.cpp:298] Extension watcher ending: osquery core has gone away
10:34 AM
when ran with normal user, it generated the expected output but i still see its process running after quitting osquery.
osquery_src/network_monitor.ext --socket /home/user/.osquery/shell.em --timeout 3 --interval 3
Mike Myers

Mike Myers

11/13/2020, 2:17 AM
It sounds like there might be a bug if the extension does not exit when it should. I've opened an issue to track this. https://github.com/trailofbits/osquery-extensions/issues/62
2:20 AM
You are saying that when run in an Administrator shell,
network_monitor
produces no results for a query to
dns_events
, but it produces results when run as a standard user? @alessandrogario do you know if this is expected behavior? It seems to me like if anything, it would be the opposite.
2:28 AM
i only see these tables i guess
Did you set the environment variable to only build certain extensions? If you built them all, you should see many additional tables when loading the extension. https://github.com/trailofbits/osquery-extensions#specifying-the-extensions-to-be-built
2:44 AM
Nevermind, I see the same thing. When loading the bundled extension,
dns_events
is not present. I can't see the reason for this, so I'll open another issue.
2:56 AM
It's because https://github.com/trailofbits/osquery-extensions/blob/master/network_monitor/CMakeLists.txt#L36 uses
addOsqueryExtension
rather than
addOsqueryExtensionEx
although that might've been intentional since we had categorized this particular extension as 'experimental'
a

Ahmed

11/13/2020, 3:04 PM
sorry for late reply, and thanks a lot for your response. i tested again with both normal user and sudo. the extension works but it keep generating this error
Extension watcher ending: osquery core has gone away
and when its closed i never exists with osquery and keeps outputing to termial. and i have tested this with osqueryd and systemd i was getting same results.
Extension watcher failed to get extensions: Extension socket not available: /home/user/.osquery/shell.em
thanks a lot for those issues.
Mike Myers

Mike Myers

11/13/2020, 5:50 PM
you're welcome, and thanks for bringing these issues to our attention
5:03 PM
Hi Ahmed, the shutdown for
network_monitor.ext
should be fixed now, thanks again
a

Ahmed

11/17/2020, 5:03 PM
really appreciate all the efforts you did, and the time taken. thanks a lot.
2:36 PM
sorry minor question, when i ran it with root account i got this.
osqueryi --disable_extensions=false  --extension network_monitor.ext

Using a virtual database. Need help, type '.help'
osquery> I1118 09:21:42.418730   954 pcapreaderservice.cpp:238] Listening on the following IPv4 addresses: 10.235.173.65/255.255.252.0
I1118 09:21:42.418823   954 pcapreaderservice.cpp:250] Listening on the following IPv6 addresses: 0:0:fe80::250:56ff/::ffff:ffff:ffff:ffff:0:0
I1118 09:21:45.404930   947 extensions.cpp:298] Extension watcher ending: osquery core has gone away
terminating
Thrift: Wed Nov 18 09:21:48 2020 TSocket::open() connect() <Host:  Port: 0>Connection refused
and when i ran it with normal user, it worked very well and when terminated osqueryi it terminated well and printed this
terminating
Thrift: Wed Nov 18 09:24:41 2020 TSocket::open() connect() <Host:  Port: 0>Connection refused
i got the same root issue with osqeuryd (i guess because its running with root)
Nov 18 09:34:23 pmlkqt2 osqueryd[2574]: I1118 09:34:23.135928  2592 extensions.cpp:298] Extension watcher ending: osquery core has gone away
Nov 18 09:34:24 pmlkqt2 osqueryd[2574]: terminating
Nov 18 09:34:26 pmlkqt2 osqueryd[2574]: I1118 09:34:26.137401  2585 extensions.cpp:341] Extension UUID 38367 has gone away
is there anything i should note here. thanks a lot.
Mike Myers

Mike Myers

11/18/2020, 2:46 PM
This seems like a Thrift IPC problem, and if it only happens when running as root then I wonder if it's because the network_monitor extension drops its privileges and loses rights to the IPC. Does that sound like a valid theory @alessandrogario?
a

alessandrogario

11/19/2020, 5:32 PM
One thing you could try to do1. start osquery as root:
sudo osqueryd --verbose --disable_extensions=false --extensions_socket=/var/osquery/extensions_socket
2. start the extension
sudo ./network_monitor.ext --verbose --extensions_socket=/var/osquery/extensions_socket
5:33 PM
As Mike said, it could be caused by the ext dropping privileges
a

Ahmed

11/23/2020, 1:39 PM
tried to use that but the daemon kept on failing. thanks a lot your efforts. if there’re anything else i could do to get that logging would be appreciated.
Mike Myers

Mike Myers

11/23/2020, 5:40 PM
when this happens, it would maybe be useful to know the access controls on the Thrift file/socket. I bet it exists but has permissions that deny it from being opened. Just my theory