Channels
doorman
infrastructure
random
zercurity
community-feeds
fleet-dev
code-review
queryhub
apple-silicon
carving
tls
fim
goquery
zentral
aws
querycon
golang
zeek
file-carving
fuzzing
auditing-warroom
linen-dev
fleetosquery
plugins
jobs
arm-architecture
darkbytes
process-auditing
uptycs
android_tests
selfgroup
vendor-feeds
fleet
eclecticiq-polylogyx-extension
ebpf
website
core
general
macos
kolide
osctrl
extensions
foundation
sql
officehours
linux
windows
Title
e
ehrhardt
10/07/2020, 1:49 AMLooks like after I pushed a query pack change, osquery stopped using
rocksdbephemeralz
zwass
10/07/2020, 2:14 AMHow did you push the query packs?
database_pluginAlthough... I'm not even sure that config still exists?
e
ehrhardt
10/07/2020, 2:25 AMWe use puppet to push these changes. The query pack is one JSON file. Changes to the pack were mostly minimal. I am seeing this in the logs
I0925 13:47:46.259778 187797504 database.cpp:140] Resetting the database plugin: rocksdb
I0925 13:53:25.853595 187797504 database.cpp:140] Resetting the database plugin: rocksdb
I0925 13:59:04.234082 187797504 database.cpp:140] Resetting the database plugin: rocksdb
W0925 14:04:38.143973 289832384 database.cpp:77] Failed to activate database plugin "rocksdb": IO error: While lock file: /var/osquery/osquery.db/LOCK: Resource temporarily unavailable
I0925 14:05:03.339555 63148032 database.cpp:140] Resetting the database plugin: ephemeral
I0925 14:10:43.635411 63148032 database.cpp:140] Resetting the database plugin: ephemeral
I0925 14:16:06.220286 63148032 database.cpp:140] Resetting the database plugin: ephemeralt
theopolis
10/07/2020, 2:27 AMWhat version of osquery are you using? I think I might know what is happening and that is indeed unintended behavior that I was indirectly trying to fix recently.
e
ehrhardt
10/07/2020, 2:28 AM4.4.0
t
theopolis
10/07/2020, 2:28 AMEssentially osquery had under-the-hood fallback code that if another process was using the database, or for some reason the error on
database.cpp:77e
ehrhardt
10/07/2020, 2:29 AMHow do I stop it from using in memory storage?
t
theopolis
10/07/2020, 2:29 AMAnother process should not be trying to use the database in normal scenarios. And it is not possible to have multiple processes open a handle to the database.
However osquery should not silently fallback to in-memory. It should either work or not work.
(in theory)
Unfortunately the only fix I can think of is upgrading to 4.5.0 or 4.5.1
e
ehrhardt
10/07/2020, 2:32 AMHrm, so our experience is changing a query pack json seems to have caused osquery to permanently use in memory storage for about half our osx systems
Just a statement so you know it can happen 🙂
Were changes made in 4.5 that would prevent this from recurring?
t
theopolis
10/07/2020, 2:34 AMYes, specifically https://github.com/osquery/osquery/pull/6637
e
ehrhardt
10/07/2020, 2:34 AMPerfect, thank you
t
theopolis
10/07/2020, 2:37 AMI am sorry this is happening 😞
e
ehrhardt
10/07/2020, 2:41 AMI am just glad there is an easy solution