Title
#macos
e

ehrhardt

10/07/2020, 1:49 AM
Looks like after I pushed a query pack change, osquery stopped using
rocksdb
and started using
ephemeral
for pack results which means the results are not stored anywhere. Does anyone know where this setting configured?
zwass

zwass

10/07/2020, 2:14 AM
How did you push the query packs?
database_plugin
would be set in the flags or config file.
2:15 AM
Although... I'm not even sure that config still exists?
e

ehrhardt

10/07/2020, 2:25 AM
We use puppet to push these changes. The query pack is one JSON file. Changes to the pack were mostly minimal. I am seeing this in the logs
I0925 13:47:46.259778 187797504 database.cpp:140] Resetting the database plugin: rocksdb
I0925 13:53:25.853595 187797504 database.cpp:140] Resetting the database plugin: rocksdb
I0925 13:59:04.234082 187797504 database.cpp:140] Resetting the database plugin: rocksdb
W0925 14:04:38.143973 289832384 database.cpp:77] Failed to activate database plugin "rocksdb": IO error: While lock file: /var/osquery/osquery.db/LOCK: Resource temporarily unavailable
I0925 14:05:03.339555 63148032 database.cpp:140] Resetting the database plugin: ephemeral
I0925 14:10:43.635411 63148032 database.cpp:140] Resetting the database plugin: ephemeral
I0925 14:16:06.220286 63148032 database.cpp:140] Resetting the database plugin: ephemeral
theopolis

theopolis

10/07/2020, 2:27 AM
What version of osquery are you using? I think I might know what is happening and that is indeed unintended behavior that I was indirectly trying to fix recently.
e

ehrhardt

10/07/2020, 2:28 AM
4.4.0
theopolis

theopolis

10/07/2020, 2:28 AM
Essentially osquery had under-the-hood fallback code that if another process was using the database, or for some reason the error on
database.cpp:77
occurred, it would fallback to using in-memory storage.
e

ehrhardt

10/07/2020, 2:29 AM
How do I stop it from using in memory storage?
theopolis

theopolis

10/07/2020, 2:29 AM
Another process should not be trying to use the database in normal scenarios. And it is not possible to have multiple processes open a handle to the database.
2:30 AM
However osquery should not silently fallback to in-memory. It should either work or not work.
2:30 AM
(in theory)
2:31 AM
Unfortunately the only fix I can think of is upgrading to 4.5.0 or 4.5.1
e

ehrhardt

10/07/2020, 2:32 AM
Hrm, so our experience is changing a query pack json seems to have caused osquery to permanently use in memory storage for about half our osx systems
2:33 AM
Just a statement so you know it can happen 🙂
2:33 AM
Were changes made in 4.5 that would prevent this from recurring?
theopolis

theopolis

10/07/2020, 2:34 AM
e

ehrhardt

10/07/2020, 2:34 AM
Perfect, thank you
theopolis

theopolis

10/07/2020, 2:37 AM
I am sorry this is happening 😞
e

ehrhardt

10/07/2020, 2:41 AM
I am just glad there is an easy solution