Matt Uebel
09/14/2021, 7:19 PMZander Mackie
09/14/2021, 8:07 PMMatt Uebel
09/14/2021, 8:07 PMZander Mackie
09/14/2021, 8:08 PMdu -h /var/osquery/osquery.db
Matt Uebel
09/14/2021, 8:09 PMZander Mackie
09/14/2021, 8:12 PMZachary Case
09/14/2021, 8:12 PMbpf_socket_events
table eventually started to get trimmed/pruned and osqueryi also stablized around 538MB taken for it's process and stopped increasingZander Mackie
09/15/2021, 1:39 PM--events_max=50000
or --events_expiry=3600
Expiration age for evented data (in seconds), applied once the data is queried. Until an evented table is queried, its collected events are cached in backing-store. Events are only expired (i.e., removed from the table) when the evented table is queried. For example, if --events_expiry=1, then events older than 1 second will only appear in the next SELECT from the subscriber. If no SELECT occurs, those events will be saved in the backing store indefinitely or until the events_max limit is reached (see below). If, on the other hand, the table contains recent events that have not yet reached expiration age, the same table can be queried repeatedly in quick succession and the same data will continue to be present unless it had reached the expiration age when it was last queried, at which point it will be removed. 3600 seconds is the default, but if querying on an interval shorter than 3600, you may wish to lower this value to avoid retrieving duplicate events.
Matt Uebel
09/15/2021, 1:42 PMI remembered you can tune events a bit if needed:
Zander Mackie
09/15/2021, 1:42 PM*_events
tablesMatt Uebel
09/15/2021, 1:42 PMevents_*
flags set--events_max=50000
Zac would know better, but I think we had ~200,000 events in one of the tables, and that was at a point where it was still growingZachary Case
09/15/2021, 2:21 PMalessandrogario
Matt Uebel
10/02/2021, 10:51 PMalessandrogario
Zander Mackie
10/03/2021, 11:19 PMZachary Case
10/04/2021, 5:50 PMalessandrogario
Zander Mackie
10/04/2021, 8:37 PM[STD-DEV]20:34:16 root@si-i-0e48e6f2c9f015d2f /home/zmackie # osqueryi --flagfile=/etc/osquery/osquery.flags --config_path /etc/osquery/osquery.conf --disable_events=false --enable_bpf_events=true --verbose
I1004 20:35:18.236025 25045 init.cpp:357] osquery initialized [version=5.0.1-5-gd63adb159]
I1004 20:35:18.236084 25045 extensions.cpp:453] Could not autoload extensions: Cannot open file for reading: /etc/osquery/extensions.load
I1004 20:35:18.236202 25045 dispatcher.cpp:78] Adding new service: ExtensionWatcher (0x561cdd51ead8) to thread: 139672016529152 (0x561cdd51d820) in process 25045
I1004 20:35:18.236251 25045 dispatcher.cpp:78] Adding new service: ExtensionRunnerCore (0x561cdd51ee58) to thread: 139672008136448 (0x561cdd51f060) in process 25045
I1004 20:35:18.236304 25045 auto_constructed_tables.cpp:97] Removing stale ATC entries
I1004 20:35:18.236299 25057 interface.cpp:299] Extension manager service starting: /root/.osquery/shell.em
I1004 20:35:18.236738 25045 packs.cpp:177] No queries defined for pack testing
W1004 20:35:18.236768 25045 config.cpp:755] Error reading the query pack named: osquery-monitoring
I1004 20:35:18.239046 25045 virtual_table.cpp:1081] Table curl is disabled, not attaching
I1004 20:35:18.256918 25045 smbios_tables.cpp:105] Reading SMBIOS from sysfs DMI node
I1004 20:35:18.278662 25045 bpfeventpublisher.cpp:256] Initialized BPF probe for syscall fork (20)
I1004 20:35:18.284932 25045 bpfeventpublisher.cpp:256] Initialized BPF probe for syscall vfork (28)
I1004 20:35:18.291116 25045 bpfeventpublisher.cpp:256] Initialized BPF probe for syscall clone (36)
I1004 20:35:18.294901 25045 bpfeventpublisher.cpp:256] Initialized BPF probe for syscall close (44)
I1004 20:35:18.298766 25045 bpfeventpublisher.cpp:256] Initialized BPF probe for syscall dup (50)
I1004 20:35:18.302650 25045 bpfeventpublisher.cpp:256] Initialized BPF probe for syscall dup2 (56)
I1004 20:35:18.306568 25045 bpfeventpublisher.cpp:256] Initialized BPF probe for syscall dup3 (62)
I1004 20:35:18.320216 25045 bpfeventpublisher.cpp:256] Initialized BPF probe for syscall creat (71)
I1004 20:35:18.325398 25045 bpfeventpublisher.cpp:256] Initialized BPF probe for syscall mknod (77)
I1004 20:35:18.331529 25045 bpfeventpublisher.cpp:256] Initialized BPF probe for syscall mknodat (83)
I1004 20:35:18.343015 25045 bpfeventpublisher.cpp:256] Initialized BPF probe for syscall open (91)
I1004 20:35:18.348021 25045 bpfeventpublisher.cpp:256] Initialized BPF probe for syscall openat (97)
I1004 20:35:18.348151 25045 bpfeventpublisher.cpp:243] Failed to load the BPF probe for syscall openat2: Failed to open the tracepoint descriptor file: /sys/kernel/debug/tracing/events/syscalls/sys_enter_openat2/id. This syscall may not be available on this system, continuing despite the error
I1004 20:35:18.358871 25045 bpfeventpublisher.cpp:256] Initialized BPF probe for syscall socket (105)
I1004 20:35:18.363005 25045 bpfeventpublisher.cpp:256] Initialized BPF probe for syscall fcntl (111)
I1004 20:35:18.368074 25045 bpfeventpublisher.cpp:256] Initialized BPF probe for syscall connect (117)
I1004 20:35:18.373728 25045 bpfeventpublisher.cpp:256] Initialized BPF probe for syscall accept (123)
I1004 20:35:18.379179 25045 bpfeventpublisher.cpp:256] Initialized BPF probe for syscall accept4 (129)
I1004 20:35:18.384074 25045 bpfeventpublisher.cpp:256] Initialized BPF probe for syscall bind (135)
I1004 20:35:18.387928 25045 bpfeventpublisher.cpp:256] Initialized BPF probe for syscall listen (141)
I1004 20:35:18.400112 25045 bpfeventpublisher.cpp:256] Initialized BPF probe for syscall chdir (149)
I1004 20:35:18.404018 25045 bpfeventpublisher.cpp:256] Initialized BPF probe for syscall fchdir (155)
I1004 20:35:18.410305 25045 bpfeventpublisher.cpp:256] Initialized BPF probe for syscall name_to_handle_at (161)
I1004 20:35:18.415171 25045 bpfeventpublisher.cpp:256] Initialized BPF probe for syscall open_by_handle_at (167)
I1004 20:35:18.455710 25045 bpfeventpublisher.cpp:256] Initialized BPF probe for syscall execve (175)
I1004 20:35:18.490099 25045 bpfeventpublisher.cpp:256] Initialized BPF probe for syscall execveat (181)
I1004 20:35:18.533653 25045 eventfactory.cpp:156] Event publisher not enabled: auditeventpublisher: Publisher disabled via configuration
I1004 20:35:18.533687 25045 eventfactory.cpp:156] Event publisher not enabled: inotify: Publisher disabled via configuration
I1004 20:35:18.533704 25045 eventfactory.cpp:156] Event publisher not enabled: syslog: Publisher disabled via configuration
I1004 20:35:18.533785 25045 events.cpp:70] Skipping subscriber: apparmor_events: Subscriber disabled via configuration
I1004 20:35:18.533859 25045 events.cpp:70] Skipping subscriber: process_file_events: Subscriber disabled via configuration
I1004 20:35:18.533877 25045 events.cpp:70] Skipping subscriber: seccomp_events: Seccomp subscriber disabled via configuration
I1004 20:35:18.533890 25045 events.cpp:70] Skipping subscriber: selinux_events: Subscriber disabled via configuration
I1004 20:35:18.533905 25045 events.cpp:70] Skipping subscriber: socket_events: Subscriber disabled via configuration
Using a virtual database. Need help, type '.help'
I1004 20:35:18.534046 25174 eventfactory.cpp:390] Starting event publisher run loop: BPFEventPublisher
osquery> I1004 20:35:18.534104 25176 eventfactory.cpp:390] Starting event publisher run loop: udev
I1004 20:35:21.561990 25174 systemstatetracker.cpp:257] Created empty process context for pid 25069. Fields will show up empty
I1004 20:35:22.563069 25174 bpfeventpublisher.cpp:374] BPFEventPublisher failed to process event from tracer #175
I1004 20:35:22.563122 25174 bpfeventpublisher.cpp:374] BPFEventPublisher failed to process event from tracer #175
I1004 20:35:22.563146 25174 bpfeventpublisher.cpp:374] BPFEventPublisher failed to process event from tracer #175
I1004 20:35:22.563165 25174 bpfeventpublisher.cpp:374] BPFEventPublisher failed to process event from tracer #175
I1004 20:35:22.564069 25174 bpfeventpublisher.cpp:374] BPFEventPublisher failed to process event from tracer #175
I1004 20:35:22.564110 25174 bpfeventpublisher.cpp:374] BPFEventPublisher failed to process event from tracer #175
I1004 20:35:22.564128 25174 bpfeventpublisher.cpp:374] BPFEventPublisher failed to process event from tracer #175
I1004 20:35:22.564143 25174 bpfeventpublisher.cpp:374] BPFEventPublisher failed to process event from tracer #175
I1004 20:35:22.568567 25174 bpfeventpublisher.cpp:374] BPFEventPublisher failed to process event from tracer #175
I1004 20:35:22.568610 25174 bpfeventpublisher.cpp:374] BPFEventPublisher failed to process event from tracer #175
I1004 20:35:22.568632 25174 bpfeventpublisher.cpp:374] BPFEventPublisher failed to process event from tracer #175
I1004 20:35:22.568643 25174 bpfeventpublisher.cpp:374] BPFEventPublisher failed to process event from tracer #175
E1004 20:35:22.572381 25174 bpfeventpublisher.cpp:381] BPFEventPublisher has encountered 12 malformed events
alessandrogario
osqueryi --version
osqueryi version 5.0.1-9-g63e5dd6fb
Zander Mackie
10/04/2021, 9:03 PM[STD-DEV]21:05:06 root@si-i-0e48e6f2c9f015d2f /home/zmackie # osqueryi --flagfile=/etc/osquery/osquery.flags --config_path /etc/osquery/osquery.conf --disable_events=false --enable_bpf_events=true --verbose
I1004 21:05:07.955135 13113 init.cpp:357] osquery initialized [version=5.0.1-9-g63e5dd6fb]
I1004 21:05:07.955199 13113 extensions.cpp:453] Could not autoload extensions: Cannot open file for reading: /etc/osquery/extensions.load
I1004 21:05:07.955307 13113 dispatcher.cpp:78] Adding new service: ExtensionWatcher (0x5648bbc85aa8) to thread: 140594447243008 (0x5648bbc955f0) in process 13113
I1004 21:05:07.955356 13113 dispatcher.cpp:78] Adding new service: ExtensionRunnerCore (0x5648bbc8a978) to thread: 140594438850304 (0x5648bbc85bf0) in process 13113
I1004 21:05:07.955379 13113 auto_constructed_tables.cpp:97] Removing stale ATC entries
I1004 21:05:07.955463 13203 interface.cpp:299] Extension manager service starting: /root/.osquery/shell.em
I1004 21:05:07.955785 13113 packs.cpp:177] No queries defined for pack testing
W1004 21:05:07.955816 13113 config.cpp:755] Error reading the query pack named: osquery-monitoring
I1004 21:05:07.959053 13113 virtual_table.cpp:1081] Table curl is disabled, not attaching
I1004 21:05:07.975880 13113 smbios_tables.cpp:105] Reading SMBIOS from sysfs DMI node
I1004 21:05:07.999199 13113 bpfeventpublisher.cpp:263] Initialized BPF probe for syscall fork (20)
I1004 21:05:08.005453 13113 bpfeventpublisher.cpp:263] Initialized BPF probe for syscall vfork (28)
I1004 21:05:08.014268 13113 bpfeventpublisher.cpp:263] Initialized BPF probe for syscall clone (36)
I1004 21:05:08.018450 13113 bpfeventpublisher.cpp:263] Initialized BPF probe for syscall close (44)
I1004 21:05:08.022289 13113 bpfeventpublisher.cpp:263] Initialized BPF probe for syscall dup (50)
I1004 21:05:08.026479 13113 bpfeventpublisher.cpp:263] Initialized BPF probe for syscall dup2 (56)
I1004 21:05:08.032091 13113 bpfeventpublisher.cpp:263] Initialized BPF probe for syscall dup3 (62)
I1004 21:05:08.044445 13113 bpfeventpublisher.cpp:263] Initialized BPF probe for syscall creat (71)
I1004 21:05:08.049504 13113 bpfeventpublisher.cpp:263] Initialized BPF probe for syscall mknod (77)
I1004 21:05:08.054567 13113 bpfeventpublisher.cpp:263] Initialized BPF probe for syscall mknodat (83)
I1004 21:05:08.067884 13113 bpfeventpublisher.cpp:263] Initialized BPF probe for syscall open (91)
I1004 21:05:08.073886 13113 bpfeventpublisher.cpp:263] Initialized BPF probe for syscall openat (97)
I1004 21:05:08.074015 13113 bpfeventpublisher.cpp:250] Failed to load the BPF probe for syscall openat2: Failed to open the tracepoint descriptor file: /sys/kernel/debug/tracing/events/syscalls/sys_enter_openat2/id. This syscall may not be available on this system, continuing despite the error
I1004 21:05:08.085897 13113 bpfeventpublisher.cpp:263] Initialized BPF probe for syscall socket (105)
I1004 21:05:08.090232 13113 bpfeventpublisher.cpp:263] Initialized BPF probe for syscall fcntl (111)
I1004 21:05:08.095454 13113 bpfeventpublisher.cpp:263] Initialized BPF probe for syscall connect (117)
I1004 21:05:08.102764 13113 bpfeventpublisher.cpp:263] Initialized BPF probe for syscall accept (123)
I1004 21:05:08.109112 13113 bpfeventpublisher.cpp:263] Initialized BPF probe for syscall accept4 (129)
I1004 21:05:08.115283 13113 bpfeventpublisher.cpp:263] Initialized BPF probe for syscall bind (135)
I1004 21:05:08.119899 13113 bpfeventpublisher.cpp:263] Initialized BPF probe for syscall listen (141)
I1004 21:05:08.133257 13113 bpfeventpublisher.cpp:263] Initialized BPF probe for syscall chdir (149)
I1004 21:05:08.137338 13113 bpfeventpublisher.cpp:263] Initialized BPF probe for syscall fchdir (155)
I1004 21:05:08.144110 13113 bpfeventpublisher.cpp:263] Initialized BPF probe for syscall name_to_handle_at (161)
I1004 21:05:08.149477 13113 bpfeventpublisher.cpp:263] Initialized BPF probe for syscall open_by_handle_at (167)
I1004 21:05:08.193707 13113 bpfeventpublisher.cpp:263] Initialized BPF probe for syscall execve (175)
I1004 21:05:08.229805 13113 bpfeventpublisher.cpp:263] Initialized BPF probe for syscall execveat (181)
I1004 21:05:08.279366 13113 eventfactory.cpp:156] Event publisher not enabled: auditeventpublisher: Publisher disabled via configuration
I1004 21:05:08.279400 13113 eventfactory.cpp:156] Event publisher not enabled: inotify: Publisher disabled via configuration
I1004 21:05:08.279417 13113 eventfactory.cpp:156] Event publisher not enabled: syslog: Publisher disabled via configuration
I1004 21:05:08.279505 13113 events.cpp:70] Skipping subscriber: apparmor_events: Subscriber disabled via configuration
I1004 21:05:08.279568 13113 events.cpp:70] Skipping subscriber: process_file_events: Subscriber disabled via configuration
I1004 21:05:08.279589 13113 events.cpp:70] Skipping subscriber: seccomp_events: Seccomp subscriber disabled via configuration
I1004 21:05:08.279609 13113 events.cpp:70] Skipping subscriber: selinux_events: Subscriber disabled via configuration
I1004 21:05:08.279628 13113 events.cpp:70] Skipping subscriber: socket_events: Subscriber disabled via configuration
I1004 21:05:08.279786 13934 eventfactory.cpp:390] Starting event publisher run loop: BPFEventPublisher
Using a virtual database. Need help, type '.help'
I1004 21:05:08.279819 13935 eventfactory.cpp:390] Starting event publisher run loop: udev
osquery> I1004 21:05:11.584556 13934 systemstatetracker.cpp:294] Created empty process context for pid 13336. Fields will show up empty
I1004 21:05:11.584614 13934 systemstatetracker.cpp:294] Created empty process context for pid 13339. Fields will show up empty
I1004 21:05:14.021695 13934 bpferrorstate.cpp:55] Buffers/strings that could not be captured by the probe: 6
I1004 21:05:14.572054 13934 systemstatetracker.cpp:294] Created empty process context for pid 19051. Fields will show up empty
I1004 21:05:14.572099 13934 systemstatetracker.cpp:294] Created empty process context for pid 19050. Fields will show up empty
I1004 21:05:14.572144 13934 systemstatetracker.cpp:294] Created empty process context for pid 18792. Fields will show up empty
I1004 21:05:14.572163 13934 systemstatetracker.cpp:294] Created empty process context for pid 19053. Fields will show up empty
I1004 21:05:14.572185 13934 systemstatetracker.cpp:294] Created empty process context for pid 19052. Fields will show up empty
I1004 21:05:17.569274 13934 systemstatetracker.cpp:294] Created empty process context for pid 25481. Fields will show up empty
I1004 21:05:17.569336 13934 systemstatetracker.cpp:294] Created empty process context for pid 25479. Fields will show up empty
I1004 21:05:20.011721 13934 bpferrorstate.cpp:55] Buffers/strings that could not be captured by the probe: 6
I1004 21:05:20.570852 13934 systemstatetracker.cpp:294] Created empty process context for pid 32490. Fields will show up empty
I1004 21:05:20.570906 13934 systemstatetracker.cpp:294] Created empty process context for pid 32493. Fields will show up empty
I1004 21:05:20.570973 13934 systemstatetracker.cpp:294] Created empty process context for pid 32494. Fields will show up empty
I1004 21:05:22.590359 13934 systemstatetracker.cpp:294] Created empty process context for pid 31807. Fields will show up empty
I1004 21:05:22.593210 13934 systemstatetracker.cpp:294] Created empty process context for pid 31808. Fields will show up empty
I1004 21:05:23.575183 13934 systemstatetracker.cpp:294] Created empty process context for pid 6849. Fields will show up empty
I1004 21:05:23.575224 13934 systemstatetracker.cpp:294] Created empty process context for pid 6851. Fields will show up empty
Buffers/strings that could not be captured by the probe:
can be 2 thingsZander Mackie
10/04/2021, 11:12 PMalessandrogario
Raff_B
03/15/2022, 5:14 PMalessandrogario
Raff_B
03/16/2022, 9:03 AMalessandrogario