hi team, I am trying to use osquery to get contin...
# ebpf
hi team, I am trying to use osquery to get continuously bpf_socket_events. I am using osqueryi since it is already installed as part of our agent. My program starts osqueryi, and every minute sends query via stdin and grabs the result via stdout. However, I noticed the memory consumption of osqueryi starts at 200M and goes up very quickly even if I configure events_max = 2000 and events_expiry = 120. Can you help me understand why is that? Also, if there is a better way to get events from bpf_socket_events continuously with low impact on the hosting machine, I would like to learn about that. Thanks a lot
@Tal Kapon any reason you wouldn’t just use osqueryd?
osqueryi is geared more towards someone at the keyboard writing an interactive query and isn’t really designed to be queried on a regular basis
thanks. the reason I used osqueryi is that it’s already installed with our agent. Also, it’s more complicated to create logging plugin than simply read stdout. Still, I can use osqueryd if there is a significant difference. Can you elaborate how come the memory footprint is so large given events_max = 2000 and events_expiry = 120? I would like to learn and understand better.
@Tal Kapon How does memory usage increase? is it after BPF is initialized? You can pass the
option to know when the probes are installed
You can estimate how much memory should be in use; scroll down to 'Memory usage' in the following link: https://osquery.readthedocs.io/en/stable/deployment/process-auditing/#linux-process-and-socket-auditing-using-bpf
The short version is
Copy code
buffer_storage_bytes = memory_pool_count * (bpf_buffer_storage_size * 4096) * possible_cpu_count


perf_bytes = (2 ^ bpf_perf_event_array_exp) * online_cpu_count
With memory_pool_count=6, bpf_buffer_storage_size set through command line options, and possible_cpu_count/online_cpu_count based on hardware
You can see those two numbers here:
Copy code
possible_cpu_count: /sys/devices/system/cpu/possible
online_cpu_count: /sys/devices/system/cpu/online
If you are running on VMware Fusion, possible_cpu_count is set to 128 unless you disable CPU hotswapping in the .vmx file
thanks. will try to follow those calculations. the cpu count based on your directions is 0-1.