https://github.com/osquery/osquery logo
Join Slack
Powered by
Hi all. I'm trying to track down the root cause of...
# general
m
Hi all. I'm trying to track down the root cause of an issue I saw on a mac device with a recent deployment. Osquery writes out an error related to expiring events and then on restart has an error from rocksdb. The odd thing is that we use a preinstall script that deletes the osquery db before reinstalling so I wouldn't expect any old events to still be around. Specifically the change is adding a query on 
es_process_events
so not sure if those event could be queued up anywhere. After reinstalling osquery or deleting the db again it resolves the issue and it doesn't seem to come back. We have the default settings for event expiry so it seems that osquery should expire events beyond 50k too. I'll put the log snippets in the tread. Appreciate any insight. Thanks
initial error
Copy code
Running duration (h:mm:ss): 0:01:15
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
E1202 11:16:38.731441 -139063488 eventsubscriberplugin.cpp:634] Failed to expire 4960339 events due to database errors
on restart
Copy code
I1211 16:15:26.482780 -208908480 init.cpp:413] osquery initialized [version=5.12.1] I1211 16:15:26.484134 -208908480 extensions.cpp:438] Found autoloadable extension: /usr/local/cof/bin/capitalone-osquery.ext I1211 16:15:26.484194 -208908480 dispatcher.cpp:78] Adding new service: WatcherRunner (0x6000006341f8) to thread: 0x16bc27000 (0x600003b36350) in process 58264 I1211 16:15:26.484268 -208908480 rocksdb.cpp:90] Opening RocksDB handle: /usr/local/cof/var/osquery/osquery.db I1211 16:15:26.484912 1807904768 watcher.cpp:772] Created and monitoring extension child (58265): /usr/local/cof/bin/capitalone-osquery.ext I1211 16:15:26.488675 -208908480 rocksdb.cpp:67] RocksDB: [WARN] [db/db_impl/db_impl_open.cc:1846] Persisting Option File error: OK I1211 16:15:26.488786 -208908480 rocksdb.cpp:181] Rocksdb open failed (5:9) IO error: No such file or directory: While opening a file for sequentially reading: /usr/local/cof/var/osquery/osquery.db/MANIFEST-000004: No such file or directory I1211 16:15:26.694521 -208908480 rocksdb.cpp:90] Opening RocksDB handle: /usr/local/cof/var/osquery/osquery.db I1211 16:15:26.697278 -208908480 rocksdb.cpp:67] RocksDB: [WARN] [db/db_impl/db_impl_open.cc:1846] Persisting Option File error: OK I1211 16:15:26.697477 -208908480 rocksdb.cpp:181] Rocksdb open failed (5:9) IO error: No such file or directory: While opening a file for sequentially reading: /usr/local/cof/var/osquery/osquery.db/MANIFEST-000004: No such file or directory
we have the esf process query running every 60 seconds and the log error wrote about a minute after startup so I think it's probably when it cleans up the events. We didn't get 4.9M events though so not sure how the count would be so high. I haven't been able to reproduce it so it's been difficult to track down what is actually happening
9 Views
Open in Slack
PreviousNext
Powered by