https://github.com/osquery/osquery logo
Join Slack
Powered by
Hey Folks, trying to build network_monitor extensi...
# general
a
Hey Folks, trying to build network_monitor extension from trailofbits followed the build guide for osquery and their github page. keep getting fatel errors. i’m building in a fresh debian:latest docker image. any advise, i’m only interested in building the extensions and not osquery itself. thanks a lot.
m
The extensions do require all of osquery to be built along with them, so they can't be compiled separately
a
ok, that what i followed actually the compilation steps in osquery docs and their github which gave me those errors and keeps failing. any advise/help how to get a clean compilation. thanks a lot
@Mike Myers getting same failure on different debian versions and ubuntu versions. and this is the command i’m following
Copy code
apt-get update -y
apt-get install git curl cmake python3 git python3 bison flex make wget -y
pip3 install timeout_decorator thrift==0.11.0 osquery pexpect==3.3
apt install --no-install-recommends rpm binutils -y
#wget <https://github.com/osquery/osquery-toolchain/releases/download/1.1.0/osquery-toolchain-1.1.0-x86_64.tar.xz>
tar xvf osquery-toolchain-1.1.0-x86_64.tar.xz -C /usr/local

git clone <https://github.com/osquery/osquery>

git clone --recurse-submodules <https://github.com/trailofbits/osquery-extensions.git>

#wget <https://github.com/Kitware/CMake/releases/download/v3.14.6/cmake-3.14.6-Linux-x86_64.tar.gz>
tar xvf cmake-3.14.6-Linux-x86_64.tar.gz -C /usr/local --strip 1
export PATH=/usr/local/bin:$PATH

cd osquery
ln -s ../../osquery-extensions ./external/extension_trailofbits

mkdir build; cd build
cmake -DOSQUERY_TOOLCHAIN_SYSROOT=/usr/local/osquery-toolchain ..
cmake --build . -j30 # where 30 is the number of parallel build jobs
m
Yea those are weird errors, but when LLVM fails during compilation sometimes it's because your build system doesn't have enough RAM
@Stefano Bonicatti what's the recommended RAM for build systems? Like 6GB?
s
Depends on how many jobs you run. ~4GB are required to build one of the C++ files from the AWS SDK and that's the only one really that has that huge requirement. 6GB might be a safe bet for maybe up to 4 jobs.
Sometimes I run with 8GBs on my Linux VM (Ubuntu) and that's fine for 10 jobs.. but not always, also depends on what else it was doing. For a 30 jobs build I would give it at least 16GBs.
But again those are somewhat estimates.. the growth is non linear, and it highly depends on what each job is doing and how many can run really parallel.. so a bit of testing is required ^^'
m
yea using 
-j30
could be increasing your memory overhead too much and one of the parallelized jobs is failing
a
it was failing before with 
-j10
and i have enough ram, but i will take your advise and test with much lower number. thanks a lot both of you
s
@Ahmed If the error is that clang gets killed it's most likely due to memory issues. How much memory do you have?
☝️ 1
m
Yea the OpenSSL library in particular is prone to failing in low memory situations, in my experience
a
i have 16G on my mac which i’m running the docker on. and now i ran the same build with 
-j4
but still getting killed.
s
@Ahmed could you please check via 
free -m
inside the Docker image that you have all 16GB? Here https://docs.docker.com/docker-for-mac/#:~:text=Memory%3A%20By%20default%2C%20Docker%20Desktop,swap%20file%20size%20as%20needed. it seems like the default is to use only 2GB from the Host.
a
i just ran it on another system with 
-j4
and it did finish compiling. but when i tried to run the
Copy code
osqueryi --extension osquery/build/external/trailofbits_osquery_extensions.ext
i dont see the tables mentioned, so trying to figure out why 😄
i only see these tables i guess:
Copy code
iptables_ext
ip6tables_ext
and in side the build i see these files. i’m only interested to get the extension for dns. thanks a lot for taking the time to look at this.
the 
dns_events
doesnot work when i run the extension using this way
Copy code
osqueryi --allow_unsafe  --extension osquery/build/external/extension_trailofbits/trailofbits_osquery_extensions.ext
and i get this message when quitting osquery
Copy code
Thrift: Wed Nov 11 07:15:12 2020 TSocket::open() connect() <Host:  Port: 0>No such file or directory
but it works with this way and generate info/error message and doenot get killed when i quit osquery i have to explicitly run 
kill
command against it.
Copy code
osqueryi --allow_unsafe  --extension osquery/build/external/extension_trailofbits/extensions/network_monitor/network_monitor.ext
and it keep generating this
Copy code
I1111 07:16:53.047338 27998 extensions.cpp:298] Extension watcher ending: osquery core has gone away
@Stefano Bonicatti any advise why the extension doesnot terminate after quitting osquery and keeps running and outputing to the screen.
Copy code
I1111 07:16:53.047338 27998 extensions.cpp:298] Extension watcher ending: osquery core has gone away
when ran with normal user, it generated the expected output but i still see its process running after quitting osquery.
Copy code
osquery_src/network_monitor.ext --socket /home/user/.osquery/shell.em --timeout 3 --interval 3
m
It sounds like there might be a bug if the extension does not exit when it should. I've opened an issue to track this. https://github.com/trailofbits/osquery-extensions/issues/62
You are saying that when run in an Administrator shell, 
network_monitor
produces no results for a query to 
dns_events
, but it produces results when run as a standard user? @alessandrogario do you know if this is expected behavior? It seems to me like if anything, it would be the opposite.
i only see these tables i guess
Did you set the environment variable to only build certain extensions? If you built them all, you should see many additional tables when loading the extension. https://github.com/trailofbits/osquery-extensions#specifying-the-extensions-to-be-built
Nevermind, I see the same thing. When loading the bundled extension, 
dns_events
is not present. I can't see the reason for this, so I'll open another issue.
It's because https://github.com/trailofbits/osquery-extensions/blob/master/network_monitor/CMakeLists.txt#L36 uses 
addOsqueryExtension
rather than 
addOsqueryExtensionEx
although that might've been intentional since we had categorized this particular extension as 'experimental'
a
sorry for late reply, and thanks a lot for your response. i tested again with both normal user and sudo. the extension works but it keep generating this error
Copy code
Extension watcher ending: osquery core has gone away
and when its closed i never exists with osquery and keeps outputing to termial. and i have tested this with osqueryd and systemd i was getting same results.
Copy code
Extension watcher failed to get extensions: Extension socket not available: /home/user/.osquery/shell.em
thanks a lot for those issues.
m
you're welcome, and thanks for bringing these issues to our attention
Hi Ahmed, the shutdown for 
network_monitor.ext
should be fixed now, thanks again
a
really appreciate all the efforts you did, and the time taken. thanks a lot.
😄 1
sorry minor question, when i ran it with root account i got this.
Copy code
osqueryi --disable_extensions=false  --extension network_monitor.ext

Using a virtual database. Need help, type '.help'
osquery> I1118 09:21:42.418730   954 pcapreaderservice.cpp:238] Listening on the following IPv4 addresses: 10.235.173.65/255.255.252.0
I1118 09:21:42.418823   954 pcapreaderservice.cpp:250] Listening on the following IPv6 addresses: 0:0:fe80::250:56ff/::ffff:ffff:ffff:ffff:0:0
I1118 09:21:45.404930   947 extensions.cpp:298] Extension watcher ending: osquery core has gone away
terminating
Thrift: Wed Nov 18 09:21:48 2020 TSocket::open() connect() <Host:  Port: 0>Connection refused
and when i ran it with normal user, it worked very well and when terminated osqueryi it terminated well and printed this
Copy code
terminating
Thrift: Wed Nov 18 09:24:41 2020 TSocket::open() connect() <Host:  Port: 0>Connection refused
i got the same root issue with osqeuryd (i guess because its running with root)
Copy code
Nov 18 09:34:23 pmlkqt2 osqueryd[2574]: I1118 09:34:23.135928  2592 extensions.cpp:298] Extension watcher ending: osquery core has gone away
Nov 18 09:34:24 pmlkqt2 osqueryd[2574]: terminating
Nov 18 09:34:26 pmlkqt2 osqueryd[2574]: I1118 09:34:26.137401  2585 extensions.cpp:341] Extension UUID 38367 has gone away
is there anything i should note here. thanks a lot.
m
This seems like a Thrift IPC problem, and if it only happens when running as root then I wonder if it's because the network_monitor extension drops its privileges and loses rights to the IPC. Does that sound like a valid theory @alessandrogario?
a
One thing you could try to do 1. start osquery as root: 
sudo osqueryd --verbose --disable_extensions=false --extensions_socket=/var/osquery/extensions_socket
2. start the extension 
sudo ./network_monitor.ext --verbose --extensions_socket=/var/osquery/extensions_socket
👍 1
As Mike said, it could be caused by the ext dropping privileges
a
tried to use that but the daemon kept on failing. thanks a lot your efforts. if there’re anything else i could do to get that logging would be appreciated.
m
when this happens, it would maybe be useful to know the access controls on the Thrift file/socket. I bet it exists but has permissions that deny it from being opened. Just my theory
19 Views
Open in Slack
PreviousNext
Powered by