https://github.com/osquery/osquery logo
Join Slack
Powered by
:wave: Hello I'm trying to get the sample <foobar>...
# extensions
s
đź‘‹ Hello I'm trying to get the sample foobar extension to load and log data to the results log via a scheduled query. When I run 
osqueryi --verbose
and check the 
.tables
, I see that the tables are present and that the extension is loaded successfully. I can also run the sample 
select * from foobar
and get results successfully. However, when I check the 
sudo cat /var/log/osquery/osqueryd.results.log
it remains empty. When I check the INFO logs, I see errors about the table 
foobar
not found.
Copy code
E0402 19:35:09.994508 124231680 scheduler.cpp:128] Error executing scheduled query my_extension_query: no such table: foobar
What's even more strange is that if I query a didn't non-custom table, 
"SELECT hostname FROM system_info;"
, I can get the results from the table and it populates the results log. Again, the goal is to simply get the result log populated with the results from the scheduled query. Does any know know why the scheduled query can not find the table or if there's an additional step that I'm missing?
my 
/var/log/osquery.conf
Copy code
{
  "options": {
    "disable_events": false
  },
  "schedule": {
    "my_extension_query": {
      "query": "SELECT * FROM foobar;",
      //"query": "SELECT * FROM my_process_events;",
      //"query": "SELECT hostname FROM system_info;",
      "interval": 20
    }
  }
}
• I check the extensions suffix to match ext • checked the directory and extensions file/owner permissions • checked that the python code runs as root
s
osqueryi and osqueryd are not really connected. How are you starting this extension, and is it connected to osqueryd?
s
đź‘‹ hello @seph In each python extension i have something similar to this;
Copy code
if __name__ == "__main__":
    osquery.start_extension(name="my_process_ext", version="1.0.0")
I've added the path of these extensions that ive tried all to 
/var/osquery/extensions.load
In 
/var/log/osquery.flags
I've added:
Copy code
--extensions_autoload=/var/osquery/extensions.load
--verbose
When i start the osquery service i've just been running 
sudo osqueryctl start
or restart.
How can i check that its connected to osqueryd and am i missing a step to connect otherwise?
s
/var/log/osquery.flags
seems odd, that should be in etc?
extensions_autoload
is the right option here.
s
its on macos . so i thought it had to be in /var where everything else is
sorry.. not log
s
logs is for logs. I forget what the path is.
s
its in 
/var/osquery
s
When I’m debugging 
osqueryd
I usually try to do things in the following order: 1. Is my config and flags syntactically correct? You can point 
osqueryi
at them 2. Does osqueryd log any errors or warnings? 3. Connect directly to the running 
osqueryd
(Either via the distributed read TLS interface, or using 
osqueryi --connect <socket path>
and debug. For (3) look at things like: • 
select * from osquery_flags
to see if it picked up the flags you’re setting • 
select * from osquery_extensions
• 
select * from osquery_registry where owner_uuid != 0;
to see what extensions did
s
Copy code
Using a virtual database. Need help, type '.help'
[*]osquery> select * from osquery_flags
[*]    ...> ;
Error: Extension call failed: connect() failed: No such file or directory
[*]osquery> select * from osquery_flags;
that seems odd because i can see the table when i run 
.tables
s
That snippet is small. That looks like a osqueryi pointed at a socket, but there’s nothing on the other end
.tables
doen’t use the socket. It doesn’t count
Copy code
$osqueryi  --connect /tmp/no/socket/here
Connected to extension socket /tmp/no/socket/here for debugging
Using a virtual database. Need help, type '.help'
[*]osquery> select * from osquery_flags;
Error: Extension call failed: connect() failed: No such file or directory
s
got it. what should i point it at for the 
<socket path>
in your example?
s
Uh, that depends on how it’s running. You’d want to check the docs for what the default is. Or use 
ps
to see how osquery invoked your extension
s
okay got it.
so running the first command is working now
and it looks like the extensions are loaded.
Copy code
| 0          |
| events_max                              | uint64 | Maximum number of event batches per type to buffer                                                                          | 50000                                   | 50000
                                                | 0          |
| events_optimize                         | bool   | Optimize subscriber select queries (scheduler only)                                                                         | true                                    | true
                                                | 0          |
| experiment_list                         | string | Comma-separated list of experiments to enable                                                                               |                                         |
                                                | 0          |
| extension                               | string | Path to a single extension to autoload                                                                                      |                                         |
                                                | 1          |
| extensions_autoload                     | string | Optional path to a list of autoloaded & managed extensions                                                                  | /var/osquery/extensions.load            | /var/osquery/extensions.
load                                            | 0          |
| extensions_default_index                | bool   | Enable INDEX on all extension table columns (default true)                                                                  | true                                    | true
                                                | 0          |
| extensions_interval                     | string | Seconds delay between connectivity checks                                                                                   | 3                                       | 3
                                                | 0          |
s
What’s 
select * from osquery_extensions
say?
s
Copy code
+-----------------------------------------+--------+-----------------------------------------------------------------------------------------------------------------------------+-----------------------------------------+-------------------------------------------------------------------------+------------+
osquery> select * from osquery_extensions;
+-------+----------------------+---------+-------------+-------------------------------------------------------------------------------+-----------+
| uuid  | name                 | version | sdk_version | path                                                                          | type      |
+-------+----------------------+---------+-------------+-------------------------------------------------------------------------------+-----------+
| 0     | core                 | 5.8.2   | 0.0.0       | /var/folders/2g/jjj_bgln5qn8c6t1szz_fmmc0000gn/T/osquery-24593/shell.em       | core      |
| 52910 | my_awesome_extension | 1.0.0   | 1.8.0       | /var/folders/2g/jjj_bgln5qn8c6t1szz_fmmc0000gn/T/osquery-24593/shell.em.52910 | extension |
| 56368 | my_process_ext       | 1.0.0   | 1.8.0       | /var/folders/2g/jjj_bgln5qn8c6t1szz_fmmc0000gn/T/osquery-24593/shell.em.56368 | extension |
+-------+----------------------+---------+-------------+-------------------------------------------------------------------------------+-----------+
osquery>
s
osquery>
is an osqueryi prompt, not connecting to osqueryd
s
Copy code
osquery> select * from osquery_registry where owner_uuid != 0;
+----------+-------------------+------------+----------+--------+
| registry | name              | owner_uuid | internal | active |
+----------+-------------------+------------+----------+--------+
| table    | foobar            | 52910      | 0        | 1      |
| table    | my_process_events | 56368      | 0        | 1      |
+----------+-------------------+------------+----------+--------+
osquery>
got it. hmm
s
Invoking and using osqueryi outside osqueryd works, but won’t really debug osqueryd. As said, you can use it’s 
--connect
to talk to the osqueryd socket. Or you can use the TLS interface.
s
gotcha. looking up the right command to connect.
so according to ls and ps
Copy code
ls -la `which osqueryi`
lrwxr-xr-x  1 root  wheel  52 Apr  1 09:06 /usr/local/bin/osqueryi -> /opt/osquery/lib/osquery.app/Contents/MacOS/osqueryd
Copy code
s -el |grep osquery
    0 76223     1     4004   0  20 10 35142996  10452 -      SNs                 0 ??         0:00.25 /opt/osquery/lib/osquery.app/Contents/MacOS/osqueryd --flagfile=/private/var/osquery/osquery.flags
    0 76224 76223     4044   0  20 10 35165200  14800 -      SN                  0 ??         0:00.10 /opt/osquery/lib/osquery.app/Contents/MacOS/osqueryd
  501 76603 17604     4006   0  31  0 34271284    968 -      R+                  0 ttys007    0:00.00 grep osq
based on the docs it looks like i can use osqueryi to .connect
s
If you want to know what socket osqueryd is using, you’d need to use ps to look at the extension, not osqueryd. And yes. you can use osqueryi to connect to osqueryd, but you need to find the socket it’s using first.
s
from ps
Copy code
77351 ttys006    0:00.08 /usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/Resources/Python.app/Contents/MacOS/Python /Users/ec2-user/osqdev/my_ext/demo.ext --socket /Users/ec2-user/.osquery/shell.em --timeout 3 --interval 3
which i think is the same socket i tried. i basically did : osqueryi then 
.connect /Users/ec2-user/.osquery/shell.em
s
Okay. that should work
Also that’s a weird path for a root osquery. Is this running as root, or as the 
ec2-user
user?
s
root
s
That seems a little weird. I wonder if that’s from the osqueryi
s
that might be the case, beacuse when i exited. i now only see
Copy code
ps -el |grep osquery
    0 77158     1     4004   0  20 10 35283284  10520 -      SNs                 0 ??         0:00.40 /opt/osquery/lib/osquery.app/Contents/MacOS/osqueryd --flagfile=/private/var/osquery/osquery.flags
    0 77159 77158     4004   0  20 10 35157016  14932 -      SN                  0 ??         0:00.13 /opt/osquery/lib/osquery.app/Contents/MacOS/osqueryd
s
grep osquery
isn’t going to find your extension.
s
i dont even see the extension running anymore based on the process output
okay. so the two extensions dont appear until AFTER ive started osqueryi in a different shell
s
From that, it souhds like osqueryd isn’t starting your extension. osqueryi will, but that’s not really helpful
I don’t know if your oqueryd is reading a different flags file, or if there’s some error somewhere.
the system logs for it might help
s
so i just stropped the osquery service with 
sudo osqueryctl stop
then ran osqueryd 
/opt/osquery/lib/osquery.app/Contents/MacOS/osqueryd --flagfile=/private/var/osquery/osquery.flags
and at least the errors disappeared
and it appears that i do get at least a couple results log now
so perhaps the service isnt correct somehow or osqueryctl isnt doing something right
s
I don’t use 
osqueryctl
, or even the osquery packages, so this starts being outside what I know. You could reboot, and see if things all shake out. Or try to read through how it works. 
osqueryctl
is mostly a wrapper that’s going to use the underlying macOS launchd stuff
s
right. i think thats the source of my issues. things seem okay when running the osqueryd process directly
Copy code
sudo /opt/osquery/lib/osquery.app/Contents/MacOS/osqueryd --flagfile=/private/var/osquery/osquery.flags
I0403 14:28:39.296025 455534080 init.cpp:413] osquery initialized [version=5.8.2]
I0403 14:28:39.298817 455534080 extensions.cpp:438] Found autoloadable extension: /Users/ec2-user/osqdev/my_ext/demo.ext
I0403 14:28:39.298899 455534080 extensions.cpp:438] Found autoloadable extension: /Users/ec2-user/osqdev/my_ext/my_extension.ext
I0403 14:28:39.298928 455534080 dispatcher.cpp:78] Adding new service: WatcherRunner (0x60000372c1f8) to thread: 0x7000023d2000 (0x600000a222a0) in process 80596
I0403 14:28:39.299501 37560320 watcher.cpp:674] osqueryd watcher (80596) executing worker (80597)
I0403 14:28:39.299984 37560320 watcher.cpp:726] Created and monitoring extension child (80598): /Users/ec2-user/osqdev/my_ext/demo.ext
I0403 14:28:39.300451 37560320 watcher.cpp:726] Created and monitoring extension child (80599): /Users/ec2-user/osqdev/my_ext/my_extension.ext
I0403 14:28:39.315002 70604288 init.cpp:410] osquery worker initialized [watcher=80596]
I0403 14:28:39.316332 70604288 dispatcher.cpp:78] Adding new service: WatcherWatcherRunner (0x6000029fc018) to thread: 0x7000063fc000 (0x6000010f4000) in process 80597
I0403 14:28:39.316404 70604288 rocksdb.cpp:90] Opening RocksDB handle: /var/osquery/osquery.db
I0403 14:28:39.342285 70604288 dispatcher.cpp:78] Adding new service: ExtensionWatcher (0x7fbc047048b8) to thread: 0x700006791000 (0x6000010f4090) in process 80597
I0403 14:28:39.342326 70604288 dispatcher.cpp:78] Adding new service: ExtensionRunnerCore (0x7fbc04705888) to thread: 0x700006814000 (0x6000010f4030) in process 80597
I0403 14:28:39.342355 70604288 auto_constructed_tables.cpp:99] Removing stale ATC entries
I0403 14:28:39.342363 109133824 interface.cpp:299] Extension manager service starting: /var/osquery/osquery.em
I0403 14:28:39.342998 70604288 eventfactory.cpp:156] Event publisher not enabled: endpointsecurity: EndpointSecurity is disabled via configuration
I0403 14:28:39.344350 70604288 eventfactory.cpp:156] Event publisher not enabled: endpointsecurity_fim: EndpointSecurity is disabled via configuration
I0403 14:28:39.344372 70604288 eventfactory.cpp:156] Event publisher not enabled: openbsm: Publisher disabled via configuration
I0403 14:28:39.344384 70604288 eventfactory.cpp:156] Event publisher not enabled: scnetwork: Publisher not used
I0403 14:28:39.344395 70604288 eventfactory.cpp:156] Event publisher not enabled: event_tapping: Publisher disabled via configuration
I0403 14:28:39.344743 70604288 eventsubscriberplugin.cpp:492] Found 24 events for subscriber diskarbitration.disk_events
I0403 14:28:39.345078 70604288 events.cpp:70] Skipping subscriber: socket_events: Subscriber disabled via configuration
I0403 14:28:39.361346 70604288 loader.cpp:45] No experiments selected
I0403 14:28:39.361413 70604288 main.cpp:104] Not starting the distributed query service: Distributed query service not enabled.
I0403 14:28:39.361420 109670400 eventfactory.cpp:390] Starting event publisher run loop: diskarbitration
I0403 14:28:39.361438 110206976 eventfactory.cpp:390] Starting event publisher run loop: fsevents
I0403 14:28:39.361439 110743552 eventfactory.cpp:390] Starting event publisher run loop: iokit
I0403 14:28:39.361454 70604288 dispatcher.cpp:78] Adding new service: SchedulerRunner (0x600002be00f8) to thread: 0x700006a20000 (0x6000010fc1c0) in process 80597
I0403 14:28:39.361470 111280128 scheduler.cpp:120] Executing scheduled query my_extension_query: SELECT * FROM foobar;
I0403 14:28:39.361505 110206976 eventfactory.cpp:410] Event publisher fsevents run loop terminated for reason: Publisher disabled via configuration
E0403 14:28:39.361850 111280128 scheduler.cpp:128] Error executing scheduled query my_extension_query: no such table: foobar
I0403 14:28:39.367128 111816704 interface.cpp:137] Registering extension (my_awesome_extension, 23266, version=1.0.0, sdk=3.0.7)
I0403 14:28:39.367218 112353280 interface.cpp:137] Registering extension (my_process_ext, 34893, version=1.0.0, sdk=3.0.7)
I0403 14:28:39.367337 111816704 registry_factory.cpp:114] Extension 23266 registered table plugin foobar
I0403 14:28:39.367471 112353280 registry_factory.cpp:114] Extension 34893 registered table plugin my_process_events


I0403 14:29:00.520484 111280128 scheduler.cpp:120] Executing scheduled query my_extension_query: SELECT * FROM foobar;
I0403 14:29:00.521996 111280128 query.cpp:128] Scheduled query has been updated: my_extension_query
I0403 14:29:00.522194 111280128 scheduler.cpp:201] Found results for query: my_extension_query
I0403 14:29:21.842377 111280128 scheduler.cpp:120] Executing scheduled query my_extension_query: SELECT * FROM foobar;
I0403 14:29:42.984807 111280128 scheduler.cpp:120] Executing scheduled query my_extension_query: SELECT * FROM foobar;
I0403 14:30:04.148504 111280128 scheduler.cpp:120] Executing scheduled query my_extension_query: SELECT * FROM foobar;
I0403 14:30:25.334856 111280128 scheduler.cpp:120] Executing scheduled query my_extension_query: SELECT * FROM foobar;
I0403 14:30:46.533720 111280128 scheduler.cpp:120] Executing scheduled query my_extension_query: SELECT * FROM foobar;
I0403 14:31:07.696177 111280128 scheduler.cpp:120] Executing scheduled query my_extension_query: SELECT * FROM foobar;

I0403 14:31:28.890303 111280128 scheduler.cpp:120] Executing scheduled query my_extension_query: SELECT * FROM foobar;
I0403 14:31:50.084290 111280128 scheduler.cpp:120] Executing scheduled query my_extension_query: SELECT * FROM foobar;
I0403 14:32:11.279191 111280128 scheduler.cpp:120] Executing scheduled query my_extension_query: SELECT * FROM foobar;
I0403 14:32:32.456142 111280128 scheduler.cpp:120] Executing scheduled query my_extension_query: SELECT * FROM foobar;
I0403 14:32:53.595430 111280128 scheduler.cpp:120] Executing scheduled query my_extension_query: SELECT * FROM foobar;
I0403 14:33:14.786928 111280128 scheduler.cpp:120] Executing scheduled query my_extension_query: SELECT * FROM foobar;
I0403 14:33:35.946554 111280128 scheduler.cpp:120] Executing scheduled query my_extension_query: SELECT * FROM foobar;
I0403 14:33:57.050361 111280128 scheduler.cpp:120] Executing scheduled query my_extension_query: SELECT * FROM foobar;
how do you start it ?
s
i work for Kolide, and we have our own packaging. It’s not really germane here.
s
got it. well thanks for all the pointers and questions.
d
I might be late, but I hope these commands help you. To run osquery shell with the extension
Copy code
osqueryi --extension ./external/external_extension_hello.ext
To run the extension detached Run the core
Copy code
sudo osqueryd --flagfile /etc/osquery/osquery.flags --extensions_socket ~/.osquery/shell.em --verbose 2>&1 | tee osquery-verbose.out
And then, run the extension in a different terminal
Copy code
sudo ./your_extension.ext --socket ~/.osquery/shell.em
Or use gdb (or any other debugger tool), doesn't work on windows:
Copy code
sudo gdb --args ./your_extension.ext --socket ~/.osquery/shell.em
I did not test those commands on MacOS, so you may need to change some paths
s
So i think the issue was that the osqueryctl service was not picking up the extensions somehow.
i just ran osqueryd directly with the right flags and things worked as expected
s
Osqueryctl shouldn’t do anything for extensions. It just restarts osqueryd.
s
but it does setup the launch daemon which i thought had keys for the flags file
which had the path to the extensions file
15 Views
Open in Slack
PreviousNext
Powered by