Title
#linux
Peter

Peter

08/05/2021, 2:57 PM
Hey there, I'm having a bit of a hard time tracking down where some
socket_events
may be disappearing to. On a low load development machine I've been finding that I don't appear to be entries in the
socket_events
table for some connections, whereas others are just fine. I was wondering if someone could point me in the right direction of how to debug this a bit further? ๐Ÿงต
2:58 PM
For a bit of background, the test system is Ubuntu 18.04.5 running Kernel 5.4.0-1051-aws with osquery 4.9.0. I'm using the Audit source for
socket_events
, and not eBPF and There's no auditd installed on the machine.
2:58 PM
The configuration flags in use has
audit_allow_sockets
, and
audit_allow_config
set to true.
disable_events
and
disable_audit
are set to false . I've used
auditctl -l
to get the installed rules when osquery is running and I'm able to see the osquery installed rule(s) for
connect,bind,execve,execveat
at syscall exit.
2:59 PM
I've also increased
events_max
temporarily on this machine to
500000
as a "just in case".
2:59 PM
As an example, running osqueryi using the flags above and running
curl <https://www.google.com>
I correctly see an entry in the
socket_events
a few moments later:
osquery> select path,remote_address from socket_events where remote_port = '443';
+---------------+----------------+
| path          | remote_address |
+---------------+----------------+
| /usr/bin/curl | 142.250.69.206 |
+---------------+----------------+
3:00 PM
However, if I compile a pretty generic reverse-shell which just opens a TCP socket to a remote server, spawns a shell, and attaches stdin, stderr, and stdout to the socket, I never see an entry in the
socket_events
table.
3:01 PM
What's especially strange is if I
strace
osquery and look for all
recvfrom
events, I can see an audit event for the associated
connect
syscall being sent over the audit socket to osquery. Apologies for the screen grab but it's a bit easier to visualise.
3:05 PM
Another screen grab (apologies) that shows the whole scenario. The "Victim" is establishing an outbound TCP connection on TCP/1234 to the "C2" via a compiled reverse-shell. A
curl
to Google shows up in the
socket_events
table shortly before, but the outbound connection for TCP/1234 doesn't appear to ever appear in the
socket_events
table.
3:09 PM
Does anyone have any pointers around how to best track down what's happening to these audit events? Once the events reach osquery? Is there some additional levels of debug - past
--verbose
- which might assist here or am I in "attach a debugger" territory? ๐Ÿ™‚
Stefano Bonicatti

Stefano Bonicatti

08/05/2021, 3:35 PM
@Peter you can try enabling
--audit_debug=true
Peter

Peter

08/05/2021, 3:48 PM
Ah nice, thank you for that! It appears to confirm that I'm seeing the events in the log, but not in the
socket_events
table:
I0805 15:45:11.034433  9994 auditdnetlink.cpp:778] 1300, audit(1628178308.957:370181): arch=c000003e syscall=42 success=no exit=-115 a0=3 a1=c00008402c a2=10 a3=0 items=0 ppid=10035 pid=10064 auid=4294967295 uid=998 gid=1002 euid=998 suid=998 fsuid=998 egid=1002 sgid=10
02 fsgid=1002 tty=pts6 ses=4294967295 comm="shell" exe="/home/ssm-user/shell/shell" key=(null)
Stefano Bonicatti

Stefano Bonicatti

08/05/2021, 3:52 PM
oh I see, the syscall has failed though. We discard failed syscalls
Peter

Peter

08/05/2021, 4:32 PM
Ah gotcha! It doesn't look like the socket has actually failed as connection completes without another call to
connect
. Looking at the fields in the audit message it appears that it may be non-blocking as result seems to indicate the exit code as
-115
. Looking at
errno.h
for Kernel 5.4 it appears this is
EINPROGRESS
?
Stefano Bonicatti

Stefano Bonicatti

08/05/2021, 4:42 PM
You're right, sorry I gave it a quick look. It was actually reported here https://github.com/osquery/osquery/issues/6609. We are indeed missing tracing for non-blocking calls, I don't recall specifically if there was some complication in handling them or they are simply a missing feature though.
Peter

Peter

08/05/2021, 4:44 PM
Not a bother at all, thank you for having a look all the same! What concerns me a little is it appears that Golang uses non-blocking sockets by default, since at least Go1.13 and likely before: https://github.com/golang/go/blob/go1.13/src/net/sock_cloexec.go#L21 I guess that'd mean these
connect
calls are "invisible" to osquery due to the issue mentioned above, at least for the time being? ๐Ÿ™‚
Stefano Bonicatti

Stefano Bonicatti

08/05/2021, 5:25 PM
I was discussing this with Alessandro (which has originally done the work), he will add some details about that on the issue I've linked,
Peter

Peter

08/05/2021, 5:26 PM
Ah that's fantastic, thank you for your help!
6:17 PM
I dropped a note as to my findings on that issue, as it's been a couple of days of digging so hopefully it helps someone else! ๐Ÿ˜‚
Stefano Bonicatti

Stefano Bonicatti

08/05/2021, 6:30 PM
Thank you! That's much appreciated
a

alessandrogario

08/19/2021, 3:53 PM
hey @Peter, this issue is being addressed here: https://github.com/osquery/osquery/pull/7269
Peter

Peter

08/20/2021, 9:28 AM
That's great news, thanks for the ping and the help! ๐Ÿ™‚