https://github.com/osquery/osquery logo
Join Slack
Powered by
Hey :wave:, It seems I have an issue with the `sy...
# general
a
Hey šŸ‘‹, It seems I have an issue with the 
systemd_units
table. I'm currently leveraging this table to make sure some services are running and be alerted otherwise. One of the use case is to monitor that we are 
wazuh-agent
running. However, on a couple of machines, Osquery results is not reliable: Example: Wazuh is running on this machine (using systemctl):
Copy code
systemctl status wazuh-agent
ā— wazuh-agent.service - Wazuh agent
   Loaded: loaded (/usr/lib/systemd/system/wazuh-agent.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2022-12-07 09:22:20 UTC; 19min ago
  Process: 19921 ExecStop=/usr/bin/env /var/ossec/bin/wazuh-control stop (code=exited, status=0/SUCCESS)
  Process: 19991 ExecStart=/usr/bin/env /var/ossec/bin/wazuh-control start (code=exited, status=0/SUCCESS)
   Memory: 11.3M
   CGroup: /system.slice/wazuh-agent.service
           ā”œā”€20022 /var/ossec/bin/wazuh-execd
           ā”œā”€20075 /var/ossec/bin/wazuh-agentd
           ā”œā”€20094 /var/ossec/bin/wazuh-syscheckd
           ā”œā”€20119 /var/ossec/bin/wazuh-logcollector
           ā””ā”€20138 /var/ossec/bin/wazuh-modulesd
However, Osquery does not see at all this service on the box:
Copy code
osquery> SELECT id, description, active_state, sub_state, fragment_path, user FROM systemd_units WHERE id LIKE 'wazuh%';
osquery>
This behaviour is just present on few machines - on all others, it 's working as expected. Would someone know how can I debug this? Is there a solution? Thank you.
s
Hey @Alexis, what distro are you running this on, and what version of osquery is this? One thing to try would be to add 
--verbose
flag to osquery and see if it shows anything..
a
Hi @sharvil. Thanks for the reply. My distro is 
CentOS Linux release 7.8.2003 (Core)
and Osquery version is 
osqueryi version 5.5.1
. Will run osquery in verbose mode and check if I can see anything.
s
Cool, and I am sure you are aware, but just to double check, osquery might need to be running as root to have permission to query systemd table
a
I'm running it as root already. I have just checked the above query using 
osqueryi --verbose
and cannot see any warning messages that could lead to not showing this 
wazuh-agent
service šŸ˜•.
What is strange is that sometimes it sees this service added and removed. Example:
Copy code
Dec  5 09:29:50 vmk-opsi-01 osqueryd[29884]: {"name":"systemd","hostIdentifier":"MACHINE_1","calendarTime":"Mon Dec  5 09:29:50 2022 UTC","unixTime":1670232590,"epoch":0,"counter":144,"numerics":false,"columns":{"active_state":"active","description":"Wazuh agent","fragment_path":"/usr/lib/systemd/system/wazuh-agent.service","id":"wazuh-agent.service","sub_state":"running","user":""},"action":"removed"}
Dec  6 00:25:24 vmk-opsi-01 osqueryd[29884]: {"name":"systemd","hostIdentifier":"MACHINE_1","calendarTime":"Tue Dec  6 00:25:24 2022 UTC","unixTime":1670286324,"epoch":0,"counter":165,"numerics":false,"columns":{"active_state":"active","description":"Wazuh agent","fragment_path":"/usr/lib/systemd/system/wazuh-agent.service","id":"wazuh-agent.service","sub_state":"running","user":""},"action":"added"}
Dec  6 07:38:45 vmk-opsi-01 osqueryd[29884]: {"name":"systemd","hostIdentifier":"MACHINE_1","calendarTime":"Tue Dec  6 07:38:45 2022 UTC","unixTime":1670312325,"epoch":0,"counter":177,"numerics":false,"columns":{"active_state":"active","description":"Wazuh agent","fragment_path":"/usr/lib/systemd/system/wazuh-agent.service","id":"wazuh-agent.service","sub_state":"running","user":""},"action":"removed"}
Dec  6 17:45:26 vmk-opsi-01 osqueryd[29884]: {"name":"systemd","hostIdentifier":"MACHINE_1","calendarTime":"Tue Dec  6 17:45:26 2022 UTC","unixTime":1670348726,"epoch":0,"counter":198,"numerics":false,"columns":{"active_state":"active","description":"Wazuh agent","fragment_path":"/usr/lib/systemd/system/wazuh-agent.service","id":"wazuh-agent.service","sub_state":"running","user":""},"action":"added"}
Dec  7 03:23:13 vmk-opsi-01 osqueryd[29884]: {"name":"systemd","hostIdentifier":"MACHINE_1","calendarTime":"Wed Dec  7 03:23:13 2022 UTC","unixTime":1670383393,"epoch":0,"counter":202,"numerics":false,"columns":{"active_state":"active","description":"Wazuh agent","fragment_path":"/usr/lib/systemd/system/wazuh-agent.service","id":"wazuh-agent.service","sub_state":"running","user":""},"action":"removed"}
Dec  7 04:49:53 vmk-opsi-01 osqueryd[29884]: {"name":"systemd","hostIdentifier":"MACHINE_1","calendarTime":"Wed Dec  7 04:49:53 2022 UTC","unixTime":1670388593,"epoch":0,"counter":205,"numerics":false,"columns":{"active_state":"active","description":"Wazuh agent","fragment_path":"/usr/lib/systemd/system/wazuh-agent.service","id":"wazuh-agent.service","sub_state":"running","user":""},"action":"added"}
Dec  7 05:18:47 vmk-opsi-01 osqueryd[29884]: {"name":"systemd","hostIdentifier":"MACHINE_1","calendarTime":"Wed Dec  7 05:18:47 2022 UTC","unixTime":1670390327,"epoch":0,"counter":206,"numerics":false,"columns":{"active_state":"active","description":"Wazuh agent","fragment_path":"/usr/lib/systemd/system/wazuh-agent.service","id":"wazuh-agent.service","sub_state":"running","user":""},"action":"removed"}
Dec  7 05:47:40 vmk-opsi-01 osqueryd[29884]: {"name":"systemd","hostIdentifier":"MACHINE_1","calendarTime":"Wed Dec  7 05:47:40 2022 UTC","unixTime":1670392060,"epoch":0,"counter":207,"numerics":false,"columns":{"active_state":"active","description":"Wazuh agent","fragment_path":"/usr/lib/systemd/system/wazuh-agent.service","id":"wazuh-agent.service","sub_state":"running","user":""},"action":"added"}
Dec  7 06:16:33 vmk-opsi-01 osqueryd[29884]: {"name":"systemd","hostIdentifier":"MACHINE_1","calendarTime":"Wed Dec  7 06:16:33 2022 UTC","unixTime":1670393793,"epoch":0,"counter":208,"numerics":false,"columns":{"active_state":"active","description":"Wazuh agent","fragment_path":"/usr/lib/systemd/system/wazuh-agent.service","id":"wazuh-agent.service","sub_state":"running","user":""},"action":"removed"}
Above are the logs that I have for osqueryd. I can see added and then removed like something is changed on the disk. But nothing changes šŸ¤”
s
Hrm..how are these queries run, are they scheduled?
a
Yes indeed.
s
I believe these scheduled queries are set to 
differential
mode, since there is the added/removed action..
maybe the 
snapshot
type makes more sense here
a
I have both in fact running (used for different purposes).
It was just to give you an idea about what's going on. Sometimes osqueryi or osqueryd sees the service present and sometimes not.
s
hrm, have you noticed this with any other systemd services? can you also use the 
startup_items
table to see if things are present there? just thinking out load, is it possible that something is causing the systemd service to go in the start/stop loop because of some error or something..?
a
Here is a machine where I do not see any issue:
Copy code
osquery> SELECT * FROM startup_items WHERE name LIKE '%wazuh%';
+---------------------+---------------------------------------------+------+--------------+--------------+---------+----------+
| name                | path                                        | args | type         | source       | status  | username |
+---------------------+---------------------------------------------+------+--------------+--------------+---------+----------+
| wazuh-agent         | /etc/rc.d/init.d/wazuh-agent                |      | Startup Item | /etc/init.d/ | enabled |          |
| wazuh-agent.service | /usr/lib/systemd/system/wazuh-agent.service |      | systemd unit |              | active  |          |
+---------------------+---------------------------------------------+------+--------------+--------------+---------+----------+
osquery> SELECT id, description, active_state, sub_state, fragment_path, user FROM systemd_units WHERE id LIKE 'wazuh%';
+---------------------+-------------+--------------+-----------+---------------------------------------------+------+
| id                  | description | active_state | sub_state | fragment_path                               | user |
+---------------------+-------------+--------------+-----------+---------------------------------------------+------+
| wazuh-agent.service | Wazuh agent | active       | running   | /usr/lib/systemd/system/wazuh-agent.service |      |
+---------------------+-------------+--------------+-----------+---------------------------------------------+------+
osquery>
And here the machine that has an issue:
Copy code
+-------------+------------------------------+------+--------------+--------------+---------+----------+
| name        | path                         | args | type         | source       | status  | username |
+-------------+------------------------------+------+--------------+--------------+---------+----------+
| wazuh-agent | /etc/rc.d/init.d/wazuh-agent |      | Startup Item | /etc/init.d/ | enabled |          |
+-------------+------------------------------+------+--------------+--------------+---------+----------+
osquery> SELECT id, description, active_state, sub_state, fragment_path, user FROM systemd_units WHERE id LIKE 'wazuh%';
osquery>
When looking at the systemd logs for this particular service (wazuh-agent), I do not see any errors. The service is working as expected. It really looks like Osquery does not pick it up even though it should.
s
yeah..and even more weirder is that this works on some hosts. I can't think of anything else off the top of my head right now..would you mind creating an issue on github please?
a
Sure will do. Thank you for your time.
Open in Slack
PreviousNext
Powered by