https://github.com/capsule8/capsule8/blob/master/docs/KProbes.md looks like the simple intro. It's based on kprobes. Additionally leverages cgroups so you can instrument differently at different processes.

@maestretti why ebpf and not static tracepoints for execve?

has there been any further progress with replacing the current version of socket_events with ebpf?

Latest packages for BPF support. This adds back bpf_socket_events with listen, accept, accept4, socket and connect. There are still no unit tests for this table, so expect hidden bugs!

(that's a super cool feature btw 🙂 )

would be nice to test on which kernel it starts to work

Hi folks, I'm going to give a talk "eBPF and the future of osquery on Linux" for the osquery@scale conference. Any particular questions come to mind or topics you'd like me to touch on? You can grab tix for the event at https://www.osqueryatscale.com/.

@alessandrogario is exit_code for bpf_process_events supposed to be the exit code for the process or the BPF probe? (or something else?)

i purposely ran a command that failed (return code 6) but the table showed 0. not sure if bug or im interpreting the column wrong

So a couple of additional notes on the matter 1. We should rename exit_code to syscall_exit_code or something else that better describes the meaning of that value 2. It is possible to capture the program exit code, but that requires additional tracing (exit and exit_group). If you think it could be useful, we could open a feature request/blueprint 3. In the case of Audit, when receiving SYSCALL events we discard everything if the 'success' field is not set to 'yes'. This means that failed execve/execveat should never appear under Audit. I think that BPF should follow what Audit is doing here

Is here a minimum supported OS version for I am running into issues with

4.4.0-142-generic

&&

Ubuntu 16.04.7 LTS

p:/home/superlog# osqueryi --verbose --disable_events=false --enable_bpf_events=true --events_expiry=1
I0202 16:22:35.706341   687 init.cpp:340] osquery initialized [version=4.6.0]
I0202 16:22:35.706419   687 extensions.cpp:453] Could not autoload extensions: Cannot open file for reading: /etc/osquery/extensions.load
I0202 16:22:35.706588   687 dispatcher.cpp:78] Adding new service: ExtensionWatcher (0x564027c14d58) to thread: 139977553381120 (0x564027c151e0) in process 687
I0202 16:22:35.706670   687 dispatcher.cpp:78] Adding new service: ExtensionRunnerCore (0x564027c1bbf8) to thread: 139977544988416 (0x564027c1be40) in process 687
I0202 16:22:35.706717   687 auto_constructed_tables.cpp:97] Removing stale ATC entries
I0202 16:22:35.706832   691 interface.cpp:270] Extension manager service starting: /root/.osquery/shell.em
terminating with uncaught exception of type tob::StringError
Aborted (core dumped)

Hi, some issues with ebpf event tables after 4.6 (built from 8b6de9788d2a60498219292b63852b25e88fe155):

I0303 03:03:34.795771 169479 bpfeventpublisher.cpp:310] Initialized BPF probe for syscall open (87)
I0303 03:03:34.800343 169479 bpfeventpublisher.cpp:310] Initialized BPF probe for syscall openat (93)
I0303 03:03:34.800472 169479 bpfeventpublisher.cpp:297] Failed to load the BPF probe for syscall openat2: Failed to open the tracepoint descriptor file: /sys/kernel/debug/tracing/events/syscalls/sys_enter_openat2/id. This syscall may not be available on this system, continuing despite the error
I0303 03:03:34.806805 169479 bpfeventpublisher.cpp:310] Initialized BPF probe for syscall socket (101)
...
I0303 03:03:34.884102 169479 bpfeventpublisher.cpp:310] Initialized BPF probe for syscall open_by_handle_at (163)
I0303 03:03:34.982189 169479 bpfeventpublisher.cpp:297] Failed to load the BPF probe for syscall __x64_sys_execve: The 'enter' program could not be loaded: Failed to open the Linux kernel version header: /usr/include/linux/version.h
I0303 03:03:34.982226 169479 eventfactory.cpp:156] Event publisher not enabled: BPFEventPublisher: Failed to create the function tracer: The 'enter' program could not be loaded: Failed to open the Linux kernel version header: /usr/include/linux/version.h

on kernel 5.4 with Ubuntu 20.04. Probably expected since it's not built from release tag, but the table used to work at 4.6.

Hi Team! Could please anyone explain inner working of ebpf implementation inside osquery/events/linux/bpf/ - as far as i can see, there is a system state tracker, which is also used to track several file operations (and connect them to corresponding PID). Does this file operations are necessary for process/socket tracing or they just used to enrich state/tables?

@Tal Kapon Welcome! You might need to install

linux-libc-dev

so that the header is available

https://github.com/microsoft/ebpf-for-windows interesting news in this space! Has anyone tested if the existing bpf tables for osquery work with this?

I’m trying to test bpf_process_events on a 4.14 kernel on CentOS 7.9. I noticed an old thread saying that CentOS 8 was needed, but there was some “verification info” here that I seem to meet. Any ideas? https://osquery.slack.com/archives/CADGD9ACF/p1603087276041700?thread_ts=1603042900.039300&cid=CADGD9ACF

seems like the wiki now contains version 4.11 as a requirements, but this is wrong (EDIT: the summary table is wrong, the BPF section is correct)

Can LINUX_VERSION_CODE be extracted from osquery's own vdso and used instead of /usr/include/linux/version.h? I can't install extra packages. It should be available for both architectures (https://github.com/torvalds/linux/blob/master/arch/x86/entry/vdso/vdso-note.S, https://github.com/torvalds/linux/blob/master/arch/arm64/kernel/vdso/note.S)

hello team, I encountered something that looks like a bug with bpf_socket_events. the ‘parent’ field is showing a process ID which is missing the first digit. Here is an example. Any clue? osquery> select pid,parent from bpf_socket_events where remote_address = ‘10.28.11.73’; [ {“parent”:“100965",“pid”:“2101011"} ] osquery> select pid,parent from processes where pid = ‘2101011’; [ {“parent”:“2100965",“pid”:“2101011"} ]

hi team, I am trying to use osquery to get continuously bpf_socket_events. I am using osqueryi since it is already installed as part of our agent. My program starts osqueryi, and every minute sends query via stdin and grabs the result via stdout. However, I noticed the memory consumption of osqueryi starts at 200M and goes up very quickly even if I configure events_max = 2000 and events_expiry = 120. Can you help me understand why is that? Also, if there is a better way to get events from bpf_socket_events continuously with low impact on the hosting machine, I would like to learn about that. Thanks a lot

@alessandrogario I’ve been seeing this too. Giving some information:

[STD-DEV]15:26:53 root@si-i-0fe66e9061a89d90d /home/zmackie # uname -a
Linux si-i-0fe66e9061a89d90d 5.4.0-1054-aws #57~18.04.1-Ubuntu SMP Thu Jul 15 03:21:36 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
[STD-DEV]15:26:55 root@si-i-0fe66e9061a89d90d /home/zmackie # cat /etc/issue
Ubuntu 18.04.5 LTS \n \l

With

--verbose

👋 hey friends if

--disable_events=false

and

--enable_bpf_events=true

are set, will this cause osquery to use significantly more memory? currently I'm testing the 5.0.0 pre-release and am seeing osquery keeps restarting because it exceeds it's memory limit, I've upped the limit to 1G with

--watchdog_memory_limit=1000

but I am still seeing memory limits exceeded on some hosts

do the ebpf socket and process tables age out or otherwise trim records? Looking that the ever-increasing memory utilization we’ve been seeing, and we can see that those tables seem to continue to grow in record count.

Will setting

--event_max

and

--event_expiry

have any effect on BPF events?

I have an Osquery agent producing the following errors with eBPF enabled:

2022-03-23 07:53:27	
Worker returned exit status
2022-03-23 07:53:08	
Error logging the results of query: pack/test-pack/BPF_PROC_EVENTS: IOError: Bad file descriptor
2022-03-23 07:53:08	
Error adding new results to database for query pack/test-pack/BPF_PROC_EVENTS: IOError: Bad file descriptor
2022-03-23 07:52:53	
RocksDB: [ERROR] [db/db_impl/db_impl_compaction_flush.cc:2541] Waiting after background compaction error: IO error: While appending to file: /var/osquery/osquery.db/052008.sst: Bad file descriptor, Accumulated background error counts: 1
2022-03-23 07:52:53	
RocksDB: [WARN] [db/error_handler.cc:334] Background IO error IO error: While appending to file: /var/osquery/osquery.db/052008.sst: Bad file descriptor
2022-03-23 07:52:53	
RocksDB: [WARN] [db/db_impl/db_impl_compaction_flush.cc:3019] Compaction error: IO error: While appending to file: /var/osquery/osquery.db/052008.sst: Bad file descriptor

eBPF flags

#### Process Auditing ####
--disable_events=false
--enable_bpf_events=true
--events_optimize=true
--events_expiry=3600
--events_max=200000

Pack queries:

SELECT * FROM bpf_socket_events WHERE local_port != 0;

SELECT * FROM bpf_process_events;

As mentioned in the office hours notes for today, here's the PoC for the new bpf_network_events table. See https://hackmd.io/023_mOVkRi2zv49WMn5PEQ?view#Introducing-the-bpf_network_events-experiment for more information

Seeing this (https://doublepulsar.com/bpfdoor-an-active-chinese-global-surveillance-tool-54b078f1a896?gi=82b009d3a9dd) makes me wonder if anyone has thought about building a

bpf_programs

table to list the loaded bpf programs? In any case, very clever use of BPF!

Somewhat related… https://arstechnica.com/information-technology/2022/06/novel-techniques-in-never-before-seen-linux-backdoor-make-it-ultra-stealthy/ is interesting. Malware that uses BPF to filter out it’s traffic from packet capture.

“When an administrator starts any packet capture tool on the infected machine, BPF bytecode is injected into the kernel that defines which packets should be captured,” the researchers wrote. “In this process, Symbiote adds its bytecode first so it can filter out network traffic that it doesn’t want the packet-capturing software to see.”