Alexis
12/07/2022, 9:48 AMsystemd_units
table. I'm currently leveraging this table to make sure some services are running and be alerted otherwise. One of the use case is to monitor that we are wazuh-agent
running. However, on a couple of machines, Osquery results is not reliable:
Example:
Wazuh is running on this machine (using systemctl):
systemctl status wazuh-agent
ā wazuh-agent.service - Wazuh agent
Loaded: loaded (/usr/lib/systemd/system/wazuh-agent.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2022-12-07 09:22:20 UTC; 19min ago
Process: 19921 ExecStop=/usr/bin/env /var/ossec/bin/wazuh-control stop (code=exited, status=0/SUCCESS)
Process: 19991 ExecStart=/usr/bin/env /var/ossec/bin/wazuh-control start (code=exited, status=0/SUCCESS)
Memory: 11.3M
CGroup: /system.slice/wazuh-agent.service
āā20022 /var/ossec/bin/wazuh-execd
āā20075 /var/ossec/bin/wazuh-agentd
āā20094 /var/ossec/bin/wazuh-syscheckd
āā20119 /var/ossec/bin/wazuh-logcollector
āā20138 /var/ossec/bin/wazuh-modulesd
However, Osquery does not see at all this service on the box:
osquery> SELECT id, description, active_state, sub_state, fragment_path, user FROM systemd_units WHERE id LIKE 'wazuh%';
osquery>
This behaviour is just present on few machines - on all others, it 's working as expected.
Would someone know how can I debug this? Is there a solution?
Thank you.sharvil
12/07/2022, 9:58 AM--verbose
flag to osquery and see if it shows anything..Alexis
12/07/2022, 10:00 AMCentOS Linux release 7.8.2003 (Core)
and Osquery version is osqueryi version 5.5.1
. Will run osquery in verbose mode and check if I can see anything.sharvil
12/07/2022, 10:01 AMAlexis
12/07/2022, 10:02 AMosqueryi --verbose
and cannot see any warning messages that could lead to not showing this wazuh-agent
service š.Dec 5 09:29:50 vmk-opsi-01 osqueryd[29884]: {"name":"systemd","hostIdentifier":"MACHINE_1","calendarTime":"Mon Dec 5 09:29:50 2022 UTC","unixTime":1670232590,"epoch":0,"counter":144,"numerics":false,"columns":{"active_state":"active","description":"Wazuh agent","fragment_path":"/usr/lib/systemd/system/wazuh-agent.service","id":"wazuh-agent.service","sub_state":"running","user":""},"action":"removed"}
Dec 6 00:25:24 vmk-opsi-01 osqueryd[29884]: {"name":"systemd","hostIdentifier":"MACHINE_1","calendarTime":"Tue Dec 6 00:25:24 2022 UTC","unixTime":1670286324,"epoch":0,"counter":165,"numerics":false,"columns":{"active_state":"active","description":"Wazuh agent","fragment_path":"/usr/lib/systemd/system/wazuh-agent.service","id":"wazuh-agent.service","sub_state":"running","user":""},"action":"added"}
Dec 6 07:38:45 vmk-opsi-01 osqueryd[29884]: {"name":"systemd","hostIdentifier":"MACHINE_1","calendarTime":"Tue Dec 6 07:38:45 2022 UTC","unixTime":1670312325,"epoch":0,"counter":177,"numerics":false,"columns":{"active_state":"active","description":"Wazuh agent","fragment_path":"/usr/lib/systemd/system/wazuh-agent.service","id":"wazuh-agent.service","sub_state":"running","user":""},"action":"removed"}
Dec 6 17:45:26 vmk-opsi-01 osqueryd[29884]: {"name":"systemd","hostIdentifier":"MACHINE_1","calendarTime":"Tue Dec 6 17:45:26 2022 UTC","unixTime":1670348726,"epoch":0,"counter":198,"numerics":false,"columns":{"active_state":"active","description":"Wazuh agent","fragment_path":"/usr/lib/systemd/system/wazuh-agent.service","id":"wazuh-agent.service","sub_state":"running","user":""},"action":"added"}
Dec 7 03:23:13 vmk-opsi-01 osqueryd[29884]: {"name":"systemd","hostIdentifier":"MACHINE_1","calendarTime":"Wed Dec 7 03:23:13 2022 UTC","unixTime":1670383393,"epoch":0,"counter":202,"numerics":false,"columns":{"active_state":"active","description":"Wazuh agent","fragment_path":"/usr/lib/systemd/system/wazuh-agent.service","id":"wazuh-agent.service","sub_state":"running","user":""},"action":"removed"}
Dec 7 04:49:53 vmk-opsi-01 osqueryd[29884]: {"name":"systemd","hostIdentifier":"MACHINE_1","calendarTime":"Wed Dec 7 04:49:53 2022 UTC","unixTime":1670388593,"epoch":0,"counter":205,"numerics":false,"columns":{"active_state":"active","description":"Wazuh agent","fragment_path":"/usr/lib/systemd/system/wazuh-agent.service","id":"wazuh-agent.service","sub_state":"running","user":""},"action":"added"}
Dec 7 05:18:47 vmk-opsi-01 osqueryd[29884]: {"name":"systemd","hostIdentifier":"MACHINE_1","calendarTime":"Wed Dec 7 05:18:47 2022 UTC","unixTime":1670390327,"epoch":0,"counter":206,"numerics":false,"columns":{"active_state":"active","description":"Wazuh agent","fragment_path":"/usr/lib/systemd/system/wazuh-agent.service","id":"wazuh-agent.service","sub_state":"running","user":""},"action":"removed"}
Dec 7 05:47:40 vmk-opsi-01 osqueryd[29884]: {"name":"systemd","hostIdentifier":"MACHINE_1","calendarTime":"Wed Dec 7 05:47:40 2022 UTC","unixTime":1670392060,"epoch":0,"counter":207,"numerics":false,"columns":{"active_state":"active","description":"Wazuh agent","fragment_path":"/usr/lib/systemd/system/wazuh-agent.service","id":"wazuh-agent.service","sub_state":"running","user":""},"action":"added"}
Dec 7 06:16:33 vmk-opsi-01 osqueryd[29884]: {"name":"systemd","hostIdentifier":"MACHINE_1","calendarTime":"Wed Dec 7 06:16:33 2022 UTC","unixTime":1670393793,"epoch":0,"counter":208,"numerics":false,"columns":{"active_state":"active","description":"Wazuh agent","fragment_path":"/usr/lib/systemd/system/wazuh-agent.service","id":"wazuh-agent.service","sub_state":"running","user":""},"action":"removed"}
sharvil
12/07/2022, 10:06 AMAlexis
12/07/2022, 10:06 AMsharvil
12/07/2022, 10:07 AMdifferential
mode, since there is the added/removed action..snapshot
type makes more sense hereAlexis
12/07/2022, 10:09 AMsharvil
12/07/2022, 10:13 AMstartup_items
table to see if things are present there? just thinking out load, is it possible that something is causing the systemd service to go in the start/stop loop because of some error or something..?Alexis
12/07/2022, 10:18 AMosquery> SELECT * FROM startup_items WHERE name LIKE '%wazuh%';
+---------------------+---------------------------------------------+------+--------------+--------------+---------+----------+
| name | path | args | type | source | status | username |
+---------------------+---------------------------------------------+------+--------------+--------------+---------+----------+
| wazuh-agent | /etc/rc.d/init.d/wazuh-agent | | Startup Item | /etc/init.d/ | enabled | |
| wazuh-agent.service | /usr/lib/systemd/system/wazuh-agent.service | | systemd unit | | active | |
+---------------------+---------------------------------------------+------+--------------+--------------+---------+----------+
osquery> SELECT id, description, active_state, sub_state, fragment_path, user FROM systemd_units WHERE id LIKE 'wazuh%';
+---------------------+-------------+--------------+-----------+---------------------------------------------+------+
| id | description | active_state | sub_state | fragment_path | user |
+---------------------+-------------+--------------+-----------+---------------------------------------------+------+
| wazuh-agent.service | Wazuh agent | active | running | /usr/lib/systemd/system/wazuh-agent.service | |
+---------------------+-------------+--------------+-----------+---------------------------------------------+------+
osquery>
And here the machine that has an issue:
+-------------+------------------------------+------+--------------+--------------+---------+----------+
| name | path | args | type | source | status | username |
+-------------+------------------------------+------+--------------+--------------+---------+----------+
| wazuh-agent | /etc/rc.d/init.d/wazuh-agent | | Startup Item | /etc/init.d/ | enabled | |
+-------------+------------------------------+------+--------------+--------------+---------+----------+
osquery> SELECT id, description, active_state, sub_state, fragment_path, user FROM systemd_units WHERE id LIKE 'wazuh%';
osquery>
sharvil
12/07/2022, 10:24 AMAlexis
12/07/2022, 10:24 AM