https://github.com/osquery/osquery logo
#macos
Title
# macos
e

ehrhardt

10/07/2020, 1:49 AM
Looks like after I pushed a query pack change, osquery stopped using
rocksdb
and started using
ephemeral
for pack results which means the results are not stored anywhere. Does anyone know where this setting configured?
z

zwass

10/07/2020, 2:14 AM
How did you push the query packs?
database_plugin
would be set in the flags or config file.
Although... I'm not even sure that config still exists?
e

ehrhardt

10/07/2020, 2:25 AM
We use puppet to push these changes. The query pack is one JSON file. Changes to the pack were mostly minimal. I am seeing this in the logs
Copy code
I0925 13:47:46.259778 187797504 database.cpp:140] Resetting the database plugin: rocksdb
I0925 13:53:25.853595 187797504 database.cpp:140] Resetting the database plugin: rocksdb
I0925 13:59:04.234082 187797504 database.cpp:140] Resetting the database plugin: rocksdb
W0925 14:04:38.143973 289832384 database.cpp:77] Failed to activate database plugin "rocksdb": IO error: While lock file: /var/osquery/osquery.db/LOCK: Resource temporarily unavailable
I0925 14:05:03.339555 63148032 database.cpp:140] Resetting the database plugin: ephemeral
I0925 14:10:43.635411 63148032 database.cpp:140] Resetting the database plugin: ephemeral
I0925 14:16:06.220286 63148032 database.cpp:140] Resetting the database plugin: ephemeral
t

theopolis

10/07/2020, 2:27 AM
What version of osquery are you using? I think I might know what is happening and that is indeed unintended behavior that I was indirectly trying to fix recently.
e

ehrhardt

10/07/2020, 2:28 AM
4.4.0
t

theopolis

10/07/2020, 2:28 AM
Essentially osquery had under-the-hood fallback code that if another process was using the database, or for some reason the error on
database.cpp:77
occurred, it would fallback to using in-memory storage.
e

ehrhardt

10/07/2020, 2:29 AM
How do I stop it from using in memory storage?
t

theopolis

10/07/2020, 2:29 AM
Another process should not be trying to use the database in normal scenarios. And it is not possible to have multiple processes open a handle to the database.
However osquery should not silently fallback to in-memory. It should either work or not work.
(in theory)
Unfortunately the only fix I can think of is upgrading to 4.5.0 or 4.5.1
e

ehrhardt

10/07/2020, 2:32 AM
Hrm, so our experience is changing a query pack json seems to have caused osquery to permanently use in memory storage for about half our osx systems
Just a statement so you know it can happen 🙂
Were changes made in 4.5 that would prevent this from recurring?
t

theopolis

10/07/2020, 2:34 AM
e

ehrhardt

10/07/2020, 2:34 AM
Perfect, thank you
t

theopolis

10/07/2020, 2:37 AM
I am sorry this is happening 😞
e

ehrhardt

10/07/2020, 2:41 AM
I am just glad there is an easy solution