Title
#general
h

HarlanF

11/01/2022, 12:11 AM
I think there's a bug with using an extension's table as a discovery query for a pack. Details in thread.
12:13 AM
It's a python extension, and here's the lines from
sudo osqueryi --verbose
that reference it:
12:13 AM
I1101 01:06:38.005260 15364 watcher.cpp:732] Created and monitoring extension child (15369): /usr/lib/osquery/bookings_meminfo.ext
W1101 01:06:38.069183 15363 packs.cpp:326] Discovery query failed (select valueKB from bookings_meminfo where key='Hugepagesize';): no such table: bookings_meminfo
I1101 01:06:38.081598 15374 interface.cpp:137] Registering extension (meminfo, 18355, version=1.0.0, sdk=1.8.0)
I1101 01:06:38.082294 15374 registry_factory.cpp:107] Extension 18355 registered table plugin bookings_meminfo
12:14 AM
This one's kind of a nonsense discovery query, but we have a more complex one that's actually causing trouble.
12:14 AM
I get no such problem when I use one of the core tables in this manner.
12:15 AM
Also, once the prompt comes up for me, I can easily do a
select * from bookings_meminfo;
and results work fine.
12:15 AM
Ideas?
s

seph

11/01/2022, 12:27 AM
Hrm. I wonder if the discovery query is triggering before the table is loaded? (IIRC those might run and get cached) Does it change if you use the osquery flag
--extensions_require
so osquery waits for this extension?
h

HarlanF

11/01/2022, 2:47 PM
@seph, I think I tried that before, but will test it today and let you know
2:48 PM
Oh, and that's definitely my read on things from the --verbose events order above...
6:30 PM
@seph Hmm, confusing. I'm assuming that it wants the extension's filename (without path or the
.ext
at the end). That works for one of the extensions, to launch osqueryi with that
--extensions_require
parameter. But if I change the discovery query to another custom extension (we have 5 presently), and attempt require the other one (or both, comma-delimited) it doesn't work.
s

seph

11/01/2022, 8:26 PM
I would expect extensions_require to be looking for the name the extension registers itself with. Which is unrelated to the file name
h

HarlanF

11/01/2022, 8:27 PM
Hmm, seems experimentally like that's not the case. Most of ours line up 1-to-1, but not one of them. That one is the unique one that seems to work at suppressing the error.
8:31 PM
Oof, I need to go read the documentation and see what's intended for each. These things don't seem like they line up:
bookings_meminfo.ext
14:        return "bookings_meminfo"
59:    osquery.start_extension(name="meminfo", version="1.0.0")

sysconfig_bookings.ext
19:        return "bookings_puppet"
91:    osquery.start_extension(name="sysconfig_bookings", version="1.0.2")

bookings_scsi_devices.ext
14:        return "bookings_scsi_devices"
71:    osquery.start_extension(name="scsi_info", version="1.0.0")

bookings_last.ext
18:        return "bookings_last"
62:    osquery.start_extension(name="bookings_last", version="1.0.1")

bookings_lsblk.ext
14:        return "bookings_lsblk"
47:    osquery.start_extension(name="bookings_lsblk", version="1.0.0")
8:38 PM
Yeah, that may be my root cause. Will get those all ship-shape and see if this problem goes away.
8:38 PM
Thanks!!
8:44 PM
@seph are there any interdependencies between file name, table name, and extension name (the one referenced by start_extension)?
s

seph

11/01/2022, 8:56 PM
Nope! I haven’t read the code, so it’s a bit 🤷 but I’m not aware of anything. I will say that Kolide’s Launcher is pretty weird — it manages itself, and registers 2 extensions with names that have nothing to do with the file name. And osquery doesn’t know the file name anyhow. So I think you have a lot of flexibility
h

HarlanF

11/01/2022, 8:57 PM
Thanks so much; I think in covering that factoid, we've stumbled across a solid lead in trying to reduce some errors in our setup. 🙂
s

seph

11/01/2022, 8:57 PM
Happy to help!
8:58 PM
one other thing to note, is that it’s a bit of a toss up whether having 1 extension or many is “better” Combining into a single extension means fewer processes running, for example
h

HarlanF

11/01/2022, 9:00 PM
Oh, that's an interesting idea. We've got 5 because they're going after completely different things. Is your thought process that we could just mash the results together into a virtual table?
s

seph

11/01/2022, 9:00 PM
No. Just register 5 tables in one extension
h

HarlanF

11/01/2022, 9:02 PM
Oh, hmm. Have to think about that, especially regarding if one of them "dies" somehow, would the watchdog restart the whole set of them?
s

seph

11/01/2022, 9:02 PM
I have no idea how the watchdog and extensions intersect.
h

HarlanF

11/01/2022, 9:03 PM
Well, that's a cool idea. I'll backlog a ticket to play with that.
s

seph

11/01/2022, 9:04 PM
Somewhat there’s a question here aboiut what’s easiest to manage. From lots of perspectives: • software dev • extension distribution • runtime stability • runtime performance Probably no really simple answer. Though at least a few years ago we all biased towards single extension, multiple things inside it
9:04 PM
(The SDKs are based around it)
h

HarlanF

11/01/2022, 9:05 PM
Oh, wow. I've been oblivious to that.
s

seph

11/01/2022, 9:05 PM
Always something new to learn 🙂
h

HarlanF

11/01/2022, 9:06 PM
(that's standing in for :amen:)
5:29 PM
Any examples of 2+ TablePlugin within one file? I'm looking at the example, with one in it, and my assumptions are:1. gather all the unique
import
lines to the top 2. have an
@osquery.register_plugin
macro at the top of each class definition 3. have a start_extension() call for each of them down in the
name == main
section
s

seph

11/02/2022, 5:31 PM
Sounds like python? I don’t really use python. but I suspect it’s calling
REGISTER_EXTERNAL
several times.
5:31 PM
Oh, which is probably
@osquery.register_plugin
in your comment.
5:31 PM
I’d expect onle a single
start_extension()
5:31 PM
Basically, you create an extension. Then you register plugins to the extension. Then you start the extension
h

HarlanF

11/02/2022, 5:32 PM
Oh, yup. Oh. Here I was thinking that the name was the table name, but you're right. Probably just the one. Not
osquery.start_extension(name="bookings_puppet", version="1.0.3")
osquery.start_extension(name="bookings_last", version="1.0.1")
osquery.start_extension(name="bookings_lsblk", version="1.0.0")
osquery.start_extension(name="bookings_meminfo", version="1.0.1")
[...]
5:32 PM
Thus the extension name is distinct from the table names
s

seph

11/02/2022, 5:32 PM
Yeah. not that. 🙂
h

HarlanF

11/02/2022, 5:32 PM
Okay, thanks! Helped to have a fresh set of eyes!
s

seph

11/02/2022, 5:32 PM
name is the extension name, version is the extension version.
5:34 PM
You can poke around some of the osquery tables that expose this.
osquery_extensions
and
osquery_registry
h

HarlanF

11/02/2022, 5:35 PM
Okay, will do. I strongly suspect we're going to get rid of a couple classes of errors by doing this. 🤞