https://github.com/osquery/osquery logo
Powered by
Title
a

Alexis

12/07/2022, 9:48 AM
Hey šŸ‘‹, It seems I have an issue with the 
systemd_units
table. I'm currently leveraging this table to make sure some services are running and be alerted otherwise. One of the use case is to monitor that we are 
wazuh-agent
running. However, on a couple of machines, Osquery results is not reliable: Example: Wazuh is running on this machine (using systemctl): 
systemctl status wazuh-agent
ā— wazuh-agent.service - Wazuh agent
   Loaded: loaded (/usr/lib/systemd/system/wazuh-agent.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2022-12-07 09:22:20 UTC; 19min ago
  Process: 19921 ExecStop=/usr/bin/env /var/ossec/bin/wazuh-control stop (code=exited, status=0/SUCCESS)
  Process: 19991 ExecStart=/usr/bin/env /var/ossec/bin/wazuh-control start (code=exited, status=0/SUCCESS)
   Memory: 11.3M
   CGroup: /system.slice/wazuh-agent.service
           ā”œā”€20022 /var/ossec/bin/wazuh-execd
           ā”œā”€20075 /var/ossec/bin/wazuh-agentd
           ā”œā”€20094 /var/ossec/bin/wazuh-syscheckd
           ā”œā”€20119 /var/ossec/bin/wazuh-logcollector
           ā””ā”€20138 /var/ossec/bin/wazuh-modulesd
However, Osquery does not see at all this service on the box: 
osquery> SELECT id, description, active_state, sub_state, fragment_path, user FROM systemd_units WHERE id LIKE 'wazuh%';
osquery>
This behaviour is just present on few machines - on all others, it 's working as expected. Would someone know how can I debug this? Is there a solution? Thank you.
s

sharvil

12/07/2022, 9:58 AM
Hey @Alexis, what distro are you running this on, and what version of osquery is this? One thing to try would be to add 
--verbose
flag to osquery and see if it shows anything..
a

Alexis

12/07/2022, 10:00 AM
Hi @sharvil. Thanks for the reply. My distro is 
CentOS Linux release 7.8.2003 (Core)
and Osquery version is 
osqueryi version 5.5.1
. Will run osquery in verbose mode and check if I can see anything.
s

sharvil

12/07/2022, 10:01 AM
Cool, and I am sure you are aware, but just to double check, osquery might need to be running as root to have permission to query systemd table
a

Alexis

12/07/2022, 10:02 AM
I'm running it as root already. I have just checked the above query using 
osqueryi --verbose
and cannot see any warning messages that could lead to not showing this 
wazuh-agent
service šŸ˜•.
What is strange is that sometimes it sees this service added and removed. Example: 
Dec  5 09:29:50 vmk-opsi-01 osqueryd[29884]: {"name":"systemd","hostIdentifier":"MACHINE_1","calendarTime":"Mon Dec  5 09:29:50 2022 UTC","unixTime":1670232590,"epoch":0,"counter":144,"numerics":false,"columns":{"active_state":"active","description":"Wazuh agent","fragment_path":"/usr/lib/systemd/system/wazuh-agent.service","id":"wazuh-agent.service","sub_state":"running","user":""},"action":"removed"}
Dec  6 00:25:24 vmk-opsi-01 osqueryd[29884]: {"name":"systemd","hostIdentifier":"MACHINE_1","calendarTime":"Tue Dec  6 00:25:24 2022 UTC","unixTime":1670286324,"epoch":0,"counter":165,"numerics":false,"columns":{"active_state":"active","description":"Wazuh agent","fragment_path":"/usr/lib/systemd/system/wazuh-agent.service","id":"wazuh-agent.service","sub_state":"running","user":""},"action":"added"}
Dec  6 07:38:45 vmk-opsi-01 osqueryd[29884]: {"name":"systemd","hostIdentifier":"MACHINE_1","calendarTime":"Tue Dec  6 07:38:45 2022 UTC","unixTime":1670312325,"epoch":0,"counter":177,"numerics":false,"columns":{"active_state":"active","description":"Wazuh agent","fragment_path":"/usr/lib/systemd/system/wazuh-agent.service","id":"wazuh-agent.service","sub_state":"running","user":""},"action":"removed"}
Dec  6 17:45:26 vmk-opsi-01 osqueryd[29884]: {"name":"systemd","hostIdentifier":"MACHINE_1","calendarTime":"Tue Dec  6 17:45:26 2022 UTC","unixTime":1670348726,"epoch":0,"counter":198,"numerics":false,"columns":{"active_state":"active","description":"Wazuh agent","fragment_path":"/usr/lib/systemd/system/wazuh-agent.service","id":"wazuh-agent.service","sub_state":"running","user":""},"action":"added"}
Dec  7 03:23:13 vmk-opsi-01 osqueryd[29884]: {"name":"systemd","hostIdentifier":"MACHINE_1","calendarTime":"Wed Dec  7 03:23:13 2022 UTC","unixTime":1670383393,"epoch":0,"counter":202,"numerics":false,"columns":{"active_state":"active","description":"Wazuh agent","fragment_path":"/usr/lib/systemd/system/wazuh-agent.service","id":"wazuh-agent.service","sub_state":"running","user":""},"action":"removed"}
Dec  7 04:49:53 vmk-opsi-01 osqueryd[29884]: {"name":"systemd","hostIdentifier":"MACHINE_1","calendarTime":"Wed Dec  7 04:49:53 2022 UTC","unixTime":1670388593,"epoch":0,"counter":205,"numerics":false,"columns":{"active_state":"active","description":"Wazuh agent","fragment_path":"/usr/lib/systemd/system/wazuh-agent.service","id":"wazuh-agent.service","sub_state":"running","user":""},"action":"added"}
Dec  7 05:18:47 vmk-opsi-01 osqueryd[29884]: {"name":"systemd","hostIdentifier":"MACHINE_1","calendarTime":"Wed Dec  7 05:18:47 2022 UTC","unixTime":1670390327,"epoch":0,"counter":206,"numerics":false,"columns":{"active_state":"active","description":"Wazuh agent","fragment_path":"/usr/lib/systemd/system/wazuh-agent.service","id":"wazuh-agent.service","sub_state":"running","user":""},"action":"removed"}
Dec  7 05:47:40 vmk-opsi-01 osqueryd[29884]: {"name":"systemd","hostIdentifier":"MACHINE_1","calendarTime":"Wed Dec  7 05:47:40 2022 UTC","unixTime":1670392060,"epoch":0,"counter":207,"numerics":false,"columns":{"active_state":"active","description":"Wazuh agent","fragment_path":"/usr/lib/systemd/system/wazuh-agent.service","id":"wazuh-agent.service","sub_state":"running","user":""},"action":"added"}
Dec  7 06:16:33 vmk-opsi-01 osqueryd[29884]: {"name":"systemd","hostIdentifier":"MACHINE_1","calendarTime":"Wed Dec  7 06:16:33 2022 UTC","unixTime":1670393793,"epoch":0,"counter":208,"numerics":false,"columns":{"active_state":"active","description":"Wazuh agent","fragment_path":"/usr/lib/systemd/system/wazuh-agent.service","id":"wazuh-agent.service","sub_state":"running","user":""},"action":"removed"}
Above are the logs that I have for osqueryd. I can see added and then removed like something is changed on the disk. But nothing changes šŸ¤”
s

sharvil

12/07/2022, 10:06 AM
Hrm..how are these queries run, are they scheduled?
a

Alexis

12/07/2022, 10:06 AM
Yes indeed.
s

sharvil

12/07/2022, 10:07 AM
I believe these scheduled queries are set to 
differential
mode, since there is the added/removed action..
maybe the 
snapshot
type makes more sense here
a

Alexis

12/07/2022, 10:09 AM
I have both in fact running (used for different purposes).
It was just to give you an idea about what's going on. Sometimes osqueryi or osqueryd sees the service present and sometimes not.
s

sharvil

12/07/2022, 10:13 AM
hrm, have you noticed this with any other systemd services? can you also use the 
startup_items
table to see if things are present there? just thinking out load, is it possible that something is causing the systemd service to go in the start/stop loop because of some error or something..?
a

Alexis

12/07/2022, 10:18 AM
Here is a machine where I do not see any issue: 
osquery> SELECT * FROM startup_items WHERE name LIKE '%wazuh%';
+---------------------+---------------------------------------------+------+--------------+--------------+---------+----------+
| name                | path                                        | args | type         | source       | status  | username |
+---------------------+---------------------------------------------+------+--------------+--------------+---------+----------+
| wazuh-agent         | /etc/rc.d/init.d/wazuh-agent                |      | Startup Item | /etc/init.d/ | enabled |          |
| wazuh-agent.service | /usr/lib/systemd/system/wazuh-agent.service |      | systemd unit |              | active  |          |
+---------------------+---------------------------------------------+------+--------------+--------------+---------+----------+
osquery> SELECT id, description, active_state, sub_state, fragment_path, user FROM systemd_units WHERE id LIKE 'wazuh%';
+---------------------+-------------+--------------+-----------+---------------------------------------------+------+
| id                  | description | active_state | sub_state | fragment_path                               | user |
+---------------------+-------------+--------------+-----------+---------------------------------------------+------+
| wazuh-agent.service | Wazuh agent | active       | running   | /usr/lib/systemd/system/wazuh-agent.service |      |
+---------------------+-------------+--------------+-----------+---------------------------------------------+------+
osquery>
And here the machine that has an issue: 
+-------------+------------------------------+------+--------------+--------------+---------+----------+
| name        | path                         | args | type         | source       | status  | username |
+-------------+------------------------------+------+--------------+--------------+---------+----------+
| wazuh-agent | /etc/rc.d/init.d/wazuh-agent |      | Startup Item | /etc/init.d/ | enabled |          |
+-------------+------------------------------+------+--------------+--------------+---------+----------+
osquery> SELECT id, description, active_state, sub_state, fragment_path, user FROM systemd_units WHERE id LIKE 'wazuh%';
osquery>
When looking at the systemd logs for this particular service (wazuh-agent), I do not see any errors. The service is working as expected. It really looks like Osquery does not pick it up even though it should.
s

sharvil

12/07/2022, 10:24 AM
yeah..and even more weirder is that this works on some hosts. I can't think of anything else off the top of my head right now..would you mind creating an issue on github please?
a

Alexis

12/07/2022, 10:24 AM
Sure will do. Thank you for your time.
#generalJoin Slack
Powered by