hey folks, I am newbie in osquery. Our team runs o...
# general
d
hey folks, I am newbie in osquery. Our team runs osquery daemon on the computers with installed Cent OS. We have some golang app which is running in the docker container and connecting to osquery using osquery socket file. And we found the issue that after reboot of the computer we can't get records from the
hardware_events
table using our golang app, But if we run the same query via
osqueryi
- it returns such events. In the golang app it just returns empty response. This issues stops to be reproducible after I restart osquery daemon - but this trick doesn't always help. I can't see any specific errors in the logs. could somebody help to troubleshoot this issue?
s
Osqueryi is not talking to osqueryd. So it may not be a good test.
I wonder if there’s a sequencing issue. Does the go program start before osquery is ready?
d
I wonder if there’s a sequencing issue. Does the go program start before osquery is ready?
yes, it is, I’ve checked status of daemon, it’s active
Do you know can we send a command with query to unix socket file directly to check result?
Or it could be related to the osquery daemon only? I suppose it could be issue with osquery-go client, but I don’t know how to check that.
s
I don’t really understand your question. osquery listens on a unix socket and uses the thrift protocol. You can talk to it via anything that can speak thrift. There are some SDKs for various thing. The go one is pretty simple
d
I thought that this issue could appear because osquery-go client has some bug and because of that my app cannot fetch query results from hardware_events table.
Because it’s really strange that I can fetch events using osqueryi, but not via golang app But issue is not always reproducible, sometimes restart of osquery daemon helps.
s
How are you fetching with osqueryi? Because there may not be any commonality there.
d
I'm executing the same query via osqueryi and my app When I execute sql query using osqueryi it works
Copy code
osquery> select he.action,he.driver, he.model, he.model_id, he.path, he.revision,  he.serial, he.vendor, he.vendor_id, DATETIME(ROUND(he.time), 'unixepoch') as time from hardware_events he;
+--------+--------+-------+----------+----------------------+----------+-------------+--------------+-----------+---------------------+
| action | driver | model | model_id | path                 | revision | serial      | vendor       | vendor_id | time                |
+--------+--------+-------+----------+----------------------+----------+-------------+--------------+-----------+---------------------+
| remove |        |       | aaaa     | /dev/bus/usb/001/019 | 0314     | 804-932-207 |           | 11ca      | 2023-03-08 08:51:46 |
| add    | usb    |       | aaaa     | /dev/bus/usb/001/020 | 0314     | 804-932-207 |          | 11ca      | 2023-03-08 08:52:09 |
+--------+--------+-------+----------+----------------------+----------+-------------+--------------+-----------+---------------------+
osquery>
but if I run app I can see that response is empty:
Copy code
{"ContextLogTag":"/tmp/osquery/router/router.go#64","level":"info","msg":"executing: select he.action,he.driver, he.model, he.model_id, he.path, he.revision,  he.serial, he.vendor, he.vendor_id, DATETIME(ROUND(he.time), 'unixepoch') as time from hardware_events he;","time":"2023-03-08T09:10:44Z"}

{"ContextLogTag":"/tmp/osquery/router/router.go#37","level":"info","msg":"response: []","time":"2023-03-08T09:10:44Z"}
Also in error logs I can see this error for osquery daemon
Copy code
[root@uk-9769-000732762e01 osquery]# cat osqueryd.ERROR
Log file created at: 2023/03/08 09:06:55
Running on machine: <id>
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0308 09:06:55.966769 10195 udev.cpp:89] udev monitor returned invalid device: No buffer space available
I don't know could be this error related to my issue or not I found similar issue on github https://github.com/osquery/osquery/issues/7758 but it's open still
@seph do you mean that osqueryi and osqueryd are highly coupled with each other and if it events appear in the osqueryi console it doesn't mean that osqueryd could fetch them?
s
osqueryi and osqueryd are highly coupled
Sorta the opposite. They are totally separate instances of osquery, and they don’t really share runtime context. I do not know what populates hardware_events, or why it might not be configured correctly on osquieryd
d
As I can see udev is used to get these events. and sometimes I can see that issue with udev
Copy code
E0308 09:06:55.966769 10195 udev.cpp:89] udev monitor returned invalid device: No buffer space available