https://github.com/osquery/osquery logo
Title
s

Stryker0x

04/02/2023, 7:58 PM
👋 Hello I'm trying to get the sample foobar extension to load and log data to the results log via a scheduled query. When I run
osqueryi --verbose
and check the
.tables
, I see that the tables are present and that the extension is loaded successfully. I can also run the sample
select * from foobar
and get results successfully. However, when I check the
sudo cat /var/log/osquery/osqueryd.results.log
it remains empty. When I check the INFO logs, I see errors about the table
foobar
not found.
E0402 19:35:09.994508 124231680 scheduler.cpp:128] Error executing scheduled query my_extension_query: no such table: foobar
What's even more strange is that if I query a didn't non-custom table,
"SELECT hostname FROM system_info;"
, I can get the results from the table and it populates the results log. Again, the goal is to simply get the result log populated with the results from the scheduled query. Does any know know why the scheduled query can not find the table or if there's an additional step that I'm missing?
my
/var/log/osquery.conf
{
  "options": {
    "disable_events": false
  },
  "schedule": {
    "my_extension_query": {
      "query": "SELECT * FROM foobar;",
      //"query": "SELECT * FROM my_process_events;",
      //"query": "SELECT hostname FROM system_info;",
      "interval": 20
    }
  }
}
• I check the extensions suffix to match ext • checked the directory and extensions file/owner permissions • checked that the python code runs as root
s

seph

04/03/2023, 1:15 AM
osqueryi and osqueryd are not really connected. How are you starting this extension, and is it connected to osqueryd?
s

Stryker0x

04/03/2023, 12:31 PM
👋 hello @seph In each python extension i have something similar to this;
if __name__ == "__main__":
    osquery.start_extension(name="my_process_ext", version="1.0.0")
I've added the path of these extensions that ive tried all to
/var/osquery/extensions.load
In
/var/log/osquery.flags
I've added:
--extensions_autoload=/var/osquery/extensions.load
--verbose
When i start the osquery service i've just been running
sudo osqueryctl start
or restart.
How can i check that its connected to osqueryd and am i missing a step to connect otherwise?
s

seph

04/03/2023, 12:54 PM
/var/log/osquery.flags
seems odd, that should be in etc?
extensions_autoload
is the right option here.
s

Stryker0x

04/03/2023, 12:57 PM
its on macos . so i thought it had to be in /var where everything else is
sorry.. not log
s

seph

04/03/2023, 12:57 PM
logs is for logs. I forget what the path is.
s

Stryker0x

04/03/2023, 12:57 PM
its in
/var/osquery
s

seph

04/03/2023, 12:58 PM
When I’m debugging
osqueryd
I usually try to do things in the following order: 1. Is my config and flags syntactically correct? You can point
osqueryi
at them 2. Does osqueryd log any errors or warnings? 3. Connect directly to the running
osqueryd
(Either via the distributed read TLS interface, or using
osqueryi --connect <socket path>
and debug. For (3) look at things like: •
select * from osquery_flags
to see if it picked up the flags you’re setting •
select * from osquery_extensions
select * from osquery_registry where owner_uuid != 0;
to see what extensions did
s

Stryker0x

04/03/2023, 1:01 PM
Using a virtual database. Need help, type '.help'
[*]osquery> select * from osquery_flags
[*]    ...> ;
Error: Extension call failed: connect() failed: No such file or directory
[*]osquery> select * from osquery_flags;
that seems odd because i can see the table when i run
.tables
s

seph

04/03/2023, 1:02 PM
That snippet is small. That looks like a osqueryi pointed at a socket, but there’s nothing on the other end
.tables
doen’t use the socket. It doesn’t count
$osqueryi  --connect /tmp/no/socket/here
Connected to extension socket /tmp/no/socket/here for debugging
Using a virtual database. Need help, type '.help'
[*]osquery> select * from osquery_flags;
Error: Extension call failed: connect() failed: No such file or directory
s

Stryker0x

04/03/2023, 1:11 PM
got it. what should i point it at for the
<socket path>
in your example?
s

seph

04/03/2023, 1:15 PM
Uh, that depends on how it’s running. You’d want to check the docs for what the default is. Or use
ps
to see how osquery invoked your extension
s

Stryker0x

04/03/2023, 1:17 PM
okay got it.
so running the first command is working now
and it looks like the extensions are loaded.
| 0          |
| events_max                              | uint64 | Maximum number of event batches per type to buffer                                                                          | 50000                                   | 50000
                                                | 0          |
| events_optimize                         | bool   | Optimize subscriber select queries (scheduler only)                                                                         | true                                    | true
                                                | 0          |
| experiment_list                         | string | Comma-separated list of experiments to enable                                                                               |                                         |
                                                | 0          |
| extension                               | string | Path to a single extension to autoload                                                                                      |                                         |
                                                | 1          |
| extensions_autoload                     | string | Optional path to a list of autoloaded & managed extensions                                                                  | /var/osquery/extensions.load            | /var/osquery/extensions.
load                                            | 0          |
| extensions_default_index                | bool   | Enable INDEX on all extension table columns (default true)                                                                  | true                                    | true
                                                | 0          |
| extensions_interval                     | string | Seconds delay between connectivity checks                                                                                   | 3                                       | 3
                                                | 0          |
s

seph

04/03/2023, 1:19 PM
What’s
select * from osquery_extensions
say?
s

Stryker0x

04/03/2023, 1:19 PM
+-----------------------------------------+--------+-----------------------------------------------------------------------------------------------------------------------------+-----------------------------------------+-------------------------------------------------------------------------+------------+
osquery> select * from osquery_extensions;
+-------+----------------------+---------+-------------+-------------------------------------------------------------------------------+-----------+
| uuid  | name                 | version | sdk_version | path                                                                          | type      |
+-------+----------------------+---------+-------------+-------------------------------------------------------------------------------+-----------+
| 0     | core                 | 5.8.2   | 0.0.0       | /var/folders/2g/jjj_bgln5qn8c6t1szz_fmmc0000gn/T/osquery-24593/shell.em       | core      |
| 52910 | my_awesome_extension | 1.0.0   | 1.8.0       | /var/folders/2g/jjj_bgln5qn8c6t1szz_fmmc0000gn/T/osquery-24593/shell.em.52910 | extension |
| 56368 | my_process_ext       | 1.0.0   | 1.8.0       | /var/folders/2g/jjj_bgln5qn8c6t1szz_fmmc0000gn/T/osquery-24593/shell.em.56368 | extension |
+-------+----------------------+---------+-------------+-------------------------------------------------------------------------------+-----------+
osquery>
s

seph

04/03/2023, 1:19 PM
osquery>
is an osqueryi prompt, not connecting to osqueryd
s

Stryker0x

04/03/2023, 1:20 PM
osquery> select * from osquery_registry where owner_uuid != 0;
+----------+-------------------+------------+----------+--------+
| registry | name              | owner_uuid | internal | active |
+----------+-------------------+------------+----------+--------+
| table    | foobar            | 52910      | 0        | 1      |
| table    | my_process_events | 56368      | 0        | 1      |
+----------+-------------------+------------+----------+--------+
osquery>
got it. hmm
s

seph

04/03/2023, 1:21 PM
Invoking and using osqueryi outside osqueryd works, but won’t really debug osqueryd. As said, you can use it’s
--connect
to talk to the osqueryd socket. Or you can use the TLS interface.
s

Stryker0x

04/03/2023, 1:22 PM
gotcha. looking up the right command to connect.
so according to ls and ps
ls -la `which osqueryi`
lrwxr-xr-x  1 root  wheel  52 Apr  1 09:06 /usr/local/bin/osqueryi -> /opt/osquery/lib/osquery.app/Contents/MacOS/osqueryd
s -el |grep osquery
    0 76223     1     4004   0  20 10 35142996  10452 -      SNs                 0 ??         0:00.25 /opt/osquery/lib/osquery.app/Contents/MacOS/osqueryd --flagfile=/private/var/osquery/osquery.flags
    0 76224 76223     4044   0  20 10 35165200  14800 -      SN                  0 ??         0:00.10 /opt/osquery/lib/osquery.app/Contents/MacOS/osqueryd
  501 76603 17604     4006   0  31  0 34271284    968 -      R+                  0 ttys007    0:00.00 grep osq
based on the docs it looks like i can use osqueryi to .connect
s

seph

04/03/2023, 1:26 PM
If you want to know what socket osqueryd is using, you’d need to use ps to look at the extension, not osqueryd. And yes. you can use osqueryi to connect to osqueryd, but you need to find the socket it’s using first.
s

Stryker0x

04/03/2023, 1:39 PM
from ps
77351 ttys006    0:00.08 /usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/Resources/Python.app/Contents/MacOS/Python /Users/ec2-user/osqdev/my_ext/demo.ext --socket /Users/ec2-user/.osquery/shell.em --timeout 3 --interval 3
which i think is the same socket i tried. i basically did : osqueryi then
.connect /Users/ec2-user/.osquery/shell.em
s

seph

04/03/2023, 1:40 PM
Okay. that should work
Also that’s a weird path for a root osquery. Is this running as root, or as the
ec2-user
user?
s

Stryker0x

04/03/2023, 1:41 PM
root
s

seph

04/03/2023, 1:41 PM
That seems a little weird. I wonder if that’s from the osqueryi
s

Stryker0x

04/03/2023, 1:42 PM
that might be the case, beacuse when i exited. i now only see
ps -el |grep osquery
    0 77158     1     4004   0  20 10 35283284  10520 -      SNs                 0 ??         0:00.40 /opt/osquery/lib/osquery.app/Contents/MacOS/osqueryd --flagfile=/private/var/osquery/osquery.flags
    0 77159 77158     4004   0  20 10 35157016  14932 -      SN                  0 ??         0:00.13 /opt/osquery/lib/osquery.app/Contents/MacOS/osqueryd
s

seph

04/03/2023, 1:44 PM
grep osquery
isn’t going to find your extension.
s

Stryker0x

04/03/2023, 1:47 PM
i dont even see the extension running anymore based on the process output
okay. so the two extensions dont appear until AFTER ive started osqueryi in a different shell
s

seph

04/03/2023, 2:30 PM
From that, it souhds like osqueryd isn’t starting your extension. osqueryi will, but that’s not really helpful
I don’t know if your oqueryd is reading a different flags file, or if there’s some error somewhere.
the system logs for it might help
s

Stryker0x

04/03/2023, 2:32 PM
so i just stropped the osquery service with
sudo osqueryctl stop
then ran osqueryd
/opt/osquery/lib/osquery.app/Contents/MacOS/osqueryd --flagfile=/private/var/osquery/osquery.flags
and at least the errors disappeared
and it appears that i do get at least a couple results log now
so perhaps the service isnt correct somehow or osqueryctl isnt doing something right
s

seph

04/03/2023, 2:36 PM
I don’t use
osqueryctl
, or even the osquery packages, so this starts being outside what I know. You could reboot, and see if things all shake out. Or try to read through how it works.
osqueryctl
is mostly a wrapper that’s going to use the underlying macOS launchd stuff
s

Stryker0x

04/03/2023, 2:37 PM
right. i think thats the source of my issues. things seem okay when running the osqueryd process directly
sudo /opt/osquery/lib/osquery.app/Contents/MacOS/osqueryd --flagfile=/private/var/osquery/osquery.flags
I0403 14:28:39.296025 455534080 init.cpp:413] osquery initialized [version=5.8.2]
I0403 14:28:39.298817 455534080 extensions.cpp:438] Found autoloadable extension: /Users/ec2-user/osqdev/my_ext/demo.ext
I0403 14:28:39.298899 455534080 extensions.cpp:438] Found autoloadable extension: /Users/ec2-user/osqdev/my_ext/my_extension.ext
I0403 14:28:39.298928 455534080 dispatcher.cpp:78] Adding new service: WatcherRunner (0x60000372c1f8) to thread: 0x7000023d2000 (0x600000a222a0) in process 80596
I0403 14:28:39.299501 37560320 watcher.cpp:674] osqueryd watcher (80596) executing worker (80597)
I0403 14:28:39.299984 37560320 watcher.cpp:726] Created and monitoring extension child (80598): /Users/ec2-user/osqdev/my_ext/demo.ext
I0403 14:28:39.300451 37560320 watcher.cpp:726] Created and monitoring extension child (80599): /Users/ec2-user/osqdev/my_ext/my_extension.ext
I0403 14:28:39.315002 70604288 init.cpp:410] osquery worker initialized [watcher=80596]
I0403 14:28:39.316332 70604288 dispatcher.cpp:78] Adding new service: WatcherWatcherRunner (0x6000029fc018) to thread: 0x7000063fc000 (0x6000010f4000) in process 80597
I0403 14:28:39.316404 70604288 rocksdb.cpp:90] Opening RocksDB handle: /var/osquery/osquery.db
I0403 14:28:39.342285 70604288 dispatcher.cpp:78] Adding new service: ExtensionWatcher (0x7fbc047048b8) to thread: 0x700006791000 (0x6000010f4090) in process 80597
I0403 14:28:39.342326 70604288 dispatcher.cpp:78] Adding new service: ExtensionRunnerCore (0x7fbc04705888) to thread: 0x700006814000 (0x6000010f4030) in process 80597
I0403 14:28:39.342355 70604288 auto_constructed_tables.cpp:99] Removing stale ATC entries
I0403 14:28:39.342363 109133824 interface.cpp:299] Extension manager service starting: /var/osquery/osquery.em
I0403 14:28:39.342998 70604288 eventfactory.cpp:156] Event publisher not enabled: endpointsecurity: EndpointSecurity is disabled via configuration
I0403 14:28:39.344350 70604288 eventfactory.cpp:156] Event publisher not enabled: endpointsecurity_fim: EndpointSecurity is disabled via configuration
I0403 14:28:39.344372 70604288 eventfactory.cpp:156] Event publisher not enabled: openbsm: Publisher disabled via configuration
I0403 14:28:39.344384 70604288 eventfactory.cpp:156] Event publisher not enabled: scnetwork: Publisher not used
I0403 14:28:39.344395 70604288 eventfactory.cpp:156] Event publisher not enabled: event_tapping: Publisher disabled via configuration
I0403 14:28:39.344743 70604288 eventsubscriberplugin.cpp:492] Found 24 events for subscriber diskarbitration.disk_events
I0403 14:28:39.345078 70604288 events.cpp:70] Skipping subscriber: socket_events: Subscriber disabled via configuration
I0403 14:28:39.361346 70604288 loader.cpp:45] No experiments selected
I0403 14:28:39.361413 70604288 main.cpp:104] Not starting the distributed query service: Distributed query service not enabled.
I0403 14:28:39.361420 109670400 eventfactory.cpp:390] Starting event publisher run loop: diskarbitration
I0403 14:28:39.361438 110206976 eventfactory.cpp:390] Starting event publisher run loop: fsevents
I0403 14:28:39.361439 110743552 eventfactory.cpp:390] Starting event publisher run loop: iokit
I0403 14:28:39.361454 70604288 dispatcher.cpp:78] Adding new service: SchedulerRunner (0x600002be00f8) to thread: 0x700006a20000 (0x6000010fc1c0) in process 80597
I0403 14:28:39.361470 111280128 scheduler.cpp:120] Executing scheduled query my_extension_query: SELECT * FROM foobar;
I0403 14:28:39.361505 110206976 eventfactory.cpp:410] Event publisher fsevents run loop terminated for reason: Publisher disabled via configuration
E0403 14:28:39.361850 111280128 scheduler.cpp:128] Error executing scheduled query my_extension_query: no such table: foobar
I0403 14:28:39.367128 111816704 interface.cpp:137] Registering extension (my_awesome_extension, 23266, version=1.0.0, sdk=3.0.7)
I0403 14:28:39.367218 112353280 interface.cpp:137] Registering extension (my_process_ext, 34893, version=1.0.0, sdk=3.0.7)
I0403 14:28:39.367337 111816704 registry_factory.cpp:114] Extension 23266 registered table plugin foobar
I0403 14:28:39.367471 112353280 registry_factory.cpp:114] Extension 34893 registered table plugin my_process_events


I0403 14:29:00.520484 111280128 scheduler.cpp:120] Executing scheduled query my_extension_query: SELECT * FROM foobar;
I0403 14:29:00.521996 111280128 query.cpp:128] Scheduled query has been updated: my_extension_query
I0403 14:29:00.522194 111280128 scheduler.cpp:201] Found results for query: my_extension_query
I0403 14:29:21.842377 111280128 scheduler.cpp:120] Executing scheduled query my_extension_query: SELECT * FROM foobar;
I0403 14:29:42.984807 111280128 scheduler.cpp:120] Executing scheduled query my_extension_query: SELECT * FROM foobar;
I0403 14:30:04.148504 111280128 scheduler.cpp:120] Executing scheduled query my_extension_query: SELECT * FROM foobar;
I0403 14:30:25.334856 111280128 scheduler.cpp:120] Executing scheduled query my_extension_query: SELECT * FROM foobar;
I0403 14:30:46.533720 111280128 scheduler.cpp:120] Executing scheduled query my_extension_query: SELECT * FROM foobar;
I0403 14:31:07.696177 111280128 scheduler.cpp:120] Executing scheduled query my_extension_query: SELECT * FROM foobar;

I0403 14:31:28.890303 111280128 scheduler.cpp:120] Executing scheduled query my_extension_query: SELECT * FROM foobar;
I0403 14:31:50.084290 111280128 scheduler.cpp:120] Executing scheduled query my_extension_query: SELECT * FROM foobar;
I0403 14:32:11.279191 111280128 scheduler.cpp:120] Executing scheduled query my_extension_query: SELECT * FROM foobar;
I0403 14:32:32.456142 111280128 scheduler.cpp:120] Executing scheduled query my_extension_query: SELECT * FROM foobar;
I0403 14:32:53.595430 111280128 scheduler.cpp:120] Executing scheduled query my_extension_query: SELECT * FROM foobar;
I0403 14:33:14.786928 111280128 scheduler.cpp:120] Executing scheduled query my_extension_query: SELECT * FROM foobar;
I0403 14:33:35.946554 111280128 scheduler.cpp:120] Executing scheduled query my_extension_query: SELECT * FROM foobar;
I0403 14:33:57.050361 111280128 scheduler.cpp:120] Executing scheduled query my_extension_query: SELECT * FROM foobar;
how do you start it ?
s

seph

04/03/2023, 2:38 PM
i work for Kolide, and we have our own packaging. It’s not really germane here.
s

Stryker0x

04/03/2023, 2:40 PM
got it. well thanks for all the pointers and questions.
d

Daniel Bretón Suárez

04/04/2023, 1:51 PM
I might be late, but I hope these commands help you. To run osquery shell with the extension
osqueryi --extension ./external/external_extension_hello.ext
To run the extension detached Run the core
sudo osqueryd --flagfile /etc/osquery/osquery.flags --extensions_socket ~/.osquery/shell.em --verbose 2>&1 | tee osquery-verbose.out
And then, run the extension in a different terminal
sudo ./your_extension.ext --socket ~/.osquery/shell.em
Or use gdb (or any other debugger tool), doesn't work on windows:
sudo gdb --args ./your_extension.ext --socket ~/.osquery/shell.em
I did not test those commands on MacOS, so you may need to change some paths
s

Stryker0x

04/04/2023, 3:44 PM
So i think the issue was that the osqueryctl service was not picking up the extensions somehow.
i just ran osqueryd directly with the right flags and things worked as expected
s

seph

04/04/2023, 3:49 PM
Osqueryctl shouldn’t do anything for extensions. It just restarts osqueryd.
s

Stryker0x

04/04/2023, 3:50 PM
but it does setup the launch daemon which i thought had keys for the flags file
which had the path to the extensions file