Title
#ebpf
8p8c

8p8c

03/03/2021, 4:02 AM
Hi, some issues with ebpf event tables after 4.6 (built from 8b6de9788d2a60498219292b63852b25e88fe155):
I0303 03:03:34.795771 169479 bpfeventpublisher.cpp:310] Initialized BPF probe for syscall open (87)
I0303 03:03:34.800343 169479 bpfeventpublisher.cpp:310] Initialized BPF probe for syscall openat (93)
I0303 03:03:34.800472 169479 bpfeventpublisher.cpp:297] Failed to load the BPF probe for syscall openat2: Failed to open the tracepoint descriptor file: /sys/kernel/debug/tracing/events/syscalls/sys_enter_openat2/id. This syscall may not be available on this system, continuing despite the error
I0303 03:03:34.806805 169479 bpfeventpublisher.cpp:310] Initialized BPF probe for syscall socket (101)
...
I0303 03:03:34.884102 169479 bpfeventpublisher.cpp:310] Initialized BPF probe for syscall open_by_handle_at (163)
I0303 03:03:34.982189 169479 bpfeventpublisher.cpp:297] Failed to load the BPF probe for syscall __x64_sys_execve: The 'enter' program could not be loaded: Failed to open the Linux kernel version header: /usr/include/linux/version.h
I0303 03:03:34.982226 169479 eventfactory.cpp:156] Event publisher not enabled: BPFEventPublisher: Failed to create the function tracer: The 'enter' program could not be loaded: Failed to open the Linux kernel version header: /usr/include/linux/version.h
on kernel 5.4 with Ubuntu 20.04. Probably expected since it's not built from release tag, but the table used to work at 4.6.
a

alessandrogario

03/03/2021, 8:57 AM
It is failing to locate the version.h file, and can be solved by installing the linux-headers package:
sudo apt-get install linux-headers-$(uname -r)
The execve, execveat probes were converted from tracepoints to kprobes to improve accuracy; more info can be found on this comment: https://github.com/osquery/osquery/pull/6802#issuecomment-744650811 TLDR: execve tracepoints miss events sometimes, both on x86 and AArch64 (but it's worse on this arch)
8p8c

8p8c

03/03/2021, 5:53 PM
it turns out the host has linux-headers installed but /usr/include/linux/version.h does not exist.
a

alessandrogario

03/03/2021, 6:16 PM
Ok this is interesting, I thought the file was from kernel headers when in fact it's not
6:16 PM
$ apt-file search /usr/include/linux/version.h
linux-libc-dev: /usr/include/linux/version.h
6:17 PM
we don't need to include it, we only parse it as pure text
6:17 PM
this is the content (on my Ubuntu 20.04 VM):
#define LINUX_VERSION_CODE 328790
#define KERNEL_VERSION(a,b,c) (((a) << 16) + ((b) << 8) + (c))
6:18 PM
the include folder could be completely empty except for that file and bpf wouldn't mind
6:18 PM
or, if you think it could be helpful, we could also allow users to specify LINUX_VERSION_CODE with a command line flag
8p8c

8p8c

03/03/2021, 6:20 PM
perhaps the files we're looking for are in /usr/src/linux-headers-5.4.0-66-generic/include/generated/uapi/linux/version.h but with the caveat that as the linux headers files are upgraded there are many of these version.h files around. a host may have /usr/src/linux-headers-5.4.0-65-generic/include/generated/uapi/linux/version.h and /usr/src/linux-headers-5.4.0-66-generic/include/generated/uapi/linux/version.h
6:21 PM
(paths specific for ubuntu, i have not looked on other distros yet)
a

alessandrogario

03/03/2021, 6:30 PM
So I guess we could either depend on the linux headers or on the libc-dev package
6:31 PM
in the case of the linux headers, we could look for the version.h file inside "/usr/src/linux-headers-$(uname -r)"
6:42 PM
Is it possible right now to come up with a workaround on that machine to allow BPF to access that file?
6:44 PM
I'll open a new feature request issue to cover this use case; we could try to access both paths (/usr/include and /usr/src/..) to find a usable copy of the file
8p8c

8p8c

03/03/2021, 6:45 PM
for my specific case, of course. i have access to that host. however, i am thinking about more general case when different distros are used and what implications it has for deployment
a

alessandrogario

03/03/2021, 6:54 PM
Would the approach I described earlier work correctly in your opinion?
8p8c

8p8c

03/03/2021, 7:21 PM
since there have been several approaches mentioned is there a specific one you're thinking about? I see command line flag for the linux version code and scanning directory paths. The major differences with the configuration in general is if the approach is install time or app starting time or both, and how feasible these combinations are.
11:19 PM
what about pulling this out from the running kernel? (some tools appear to do that)
3:58 AM
bpftrace has had the same issue: https://github.com/iovisor/bpftrace/pull/282/