https://github.com/osquery/osquery logo
Join Slack
Powered by
Hey there, I'm having a bit of a hard time trackin...
# linux
p
Hey there, I'm having a bit of a hard time tracking down where some 
socket_events
may be disappearing to. On a low load development machine I've been finding that I don't appear to be entries in the 
socket_events
table for some connections, whereas others are just fine. I was wondering if someone could point me in the right direction of how to debug this a bit further? 🧵
For a bit of background, the test system is Ubuntu 18.04.5 running Kernel 5.4.0-1051-aws with osquery 4.9.0. I'm using the Audit source for 
socket_events
, and not eBPF and There's no auditd installed on the machine.
The configuration flags in use has 
audit_allow_sockets
, and 
audit_allow_config
set to true. 
disable_events
and 
disable_audit
are set to false . I've used 
auditctl -l
to get the installed rules when osquery is running and I'm able to see the osquery installed rule(s) for 
connect,bind,execve,execveat
at syscall exit.
I've also increased 
events_max
temporarily on this machine to 
500000
as a "just in case".
As an example, running osqueryi using the flags above and running 
curl <https://www.google.com>
I correctly see an entry in the 
socket_events
a few moments later:
Copy code
osquery> select path,remote_address from socket_events where remote_port = '443';
+---------------+----------------+
| path          | remote_address |
+---------------+----------------+
| /usr/bin/curl | 142.250.69.206 |
+---------------+----------------+
However, if I compile a pretty generic reverse-shell which just opens a TCP socket to a remote server, spawns a shell, and attaches stdin, stderr, and stdout to the socket, I never see an entry in the 
socket_events
table.
What's especially strange is if I 
strace
osquery and look for all 
recvfrom
events, I can see an audit event for the associated 
connect
syscall being sent over the audit socket to osquery. Apologies for the screen grab but it's a bit easier to visualise.
Another screen grab (apologies) that shows the whole scenario. The "Victim" is establishing an outbound TCP connection on TCP/1234 to the "C2" via a compiled reverse-shell. A 
curl
to Google shows up in the 
socket_events
table shortly before, but the outbound connection for TCP/1234 doesn't appear to ever appear in the 
socket_events
table.
Does anyone have any pointers around how to best track down what's happening to these audit events? Once the events reach osquery? Is there some additional levels of debug - past 
--verbose
- which might assist here or am I in "attach a debugger" territory? 🙂
s
@Peter you can try enabling 
--audit_debug=true
p
Ah nice, thank you for that! It appears to confirm that I'm seeing the events in the log, but not in the 
socket_events
table:
Copy code
I0805 15:45:11.034433  9994 auditdnetlink.cpp:778] 1300, audit(1628178308.957:370181): arch=c000003e syscall=42 success=no exit=-115 a0=3 a1=c00008402c a2=10 a3=0 items=0 ppid=10035 pid=10064 auid=4294967295 uid=998 gid=1002 euid=998 suid=998 fsuid=998 egid=1002 sgid=10
02 fsgid=1002 tty=pts6 ses=4294967295 comm="shell" exe="/home/ssm-user/shell/shell" key=(null)
s
oh I see, the syscall has failed though. We discard failed syscalls
p
Ah gotcha! It doesn't look like the socket has actually failed as connection completes without another call to 
connect
. Looking at the fields in the audit message it appears that it may be non-blocking as result seems to indicate the exit code as 
-115
. Looking at 
errno.h
for Kernel 5.4 it appears this is 
EINPROGRESS
?
s
You're right, sorry I gave it a quick look. It was actually reported here https://github.com/osquery/osquery/issues/6609. We are indeed missing tracing for non-blocking calls, I don't recall specifically if there was some complication in handling them or they are simply a missing feature though.
p
Not a bother at all, thank you for having a look all the same! What concerns me a little is it appears that Golang uses non-blocking sockets by default, since at least Go1.13 and likely before: https://github.com/golang/go/blob/go1.13/src/net/sock_cloexec.go#L21 I guess that'd mean these 
connect
calls are "invisible" to osquery due to the issue mentioned above, at least for the time being? 🙂
s
I was discussing this with Alessandro (which has originally done the work), he will add some details about that on the issue I've linked,
p
Ah that's fantastic, thank you for your help!
I dropped a note as to my findings on that issue, as it's been a couple of days of digging so hopefully it helps someone else! 😂
s
Thank you! That's much appreciated
a
hey @Peter, this issue is being addressed here: https://github.com/osquery/osquery/pull/7269
p
That's great news, thanks for the ping and the help! 🙂
26 Views
Open in Slack
PreviousNext
Powered by