lvferdi
10/17/2022, 6:26 PMselect timestamp as time, datetime(timestamp, 'unixepoch') AS utc_time, storage, message, actiity, process, pid, sender, tid, category, subsystem FROM unified_log WHERE (sender = 'AppleSystemPolicy' AND message like '%ASP: Security policy%') and timestamp > -1;
This tends to either get denylisted on first run or on the second. Any thoughts on how to collect this and other data from unified_log without being denylisted would be appreciated.zwass
timestamp > -1
portion? I don't think that timestamp trick was intended to be used along with the other filters, so perhaps it causes the code to hit some less optimized path.lvferdi
10/17/2022, 7:18 PMseph
lvferdi
10/17/2022, 7:52 PMseph
lvferdi
10/17/2022, 7:59 PMwatchdog_level=1
now and will see how it goes. The denylist seems to be due to number of restarts. I am getting the log
osquery worker stopping: Memory limits exceeded: 629063680
Stefano Bonicatti
10/18/2022, 11:04 AM--planner
. It should print on the top some information around how many rows are being passed behind the scenes to sqlite, pre sqlite filtering but post table logic filtering. This is likely what’s causing the high amount of memory used.
It should print something like:
osquery planner: xFilter unified_log generate returned row count:100
In my case though I see 100 rows correctly being returned; the query is almost the same as yours, I just changed the constraint on message
, because I don’t have any of those messageslvferdi
10/18/2022, 1:01 PMRyan Mack
10/18/2022, 5:08 PMlvferdi
10/18/2022, 6:23 PM--watchdog_level=0
--watchdog_memory_limit=600
Also ran the profiling script
./tools/analysis/profile.py --shell /usr/local/bin/osqueryi --config /var/osquery/osquery.conf --rounds 4 --count 10
Profiling query: SELECT timestamp as time, datetime(timestamp, 'unixepoch') AS utc_time, storage, message, activity, process, pid, sender, tid, category, subsystem, level FROM unified_log WHERE (sender = 'AppleSystemPolicy' AND message LIKE '%ASP: Security policy%');
U:0 C:0 M:2 F:0 D:0 pack_xprotect_action (1/4): utilization: 2.0 cpu_time: 0.02026575 memory: 16531456 fds: 4 duration: 0.5249452590942383
U:0 C:0 M:2 F:0 D:0 pack_xprotect_action (2/4): utilization: 2.5 cpu_time: 0.026586916 memory: 16596992 fds: 4 duration: 0.5433149337768555
U:0 C:0 M:2 F:0 D:0 pack_xprotect_action (3/4): utilization: 3.75 cpu_time: 0.038514957 memory: 17301504 fds: 4 duration: 0.5454161167144775
U:0 C:0 M:2 F:0 D:0 pack_xprotect_action (4/4): utilization: 3.6 cpu_time: 0.037013833999999995 memory: 17006592 fds: 4 duration: 0.53110671043396
U:0 C:0 M:2 F:0 D:0 pack_xprotect_action avg: utilization: 2.9625 cpu_time: 0.03059536425 memory: 16859136.0 fds: 4.0 duration: 0.5361957550048828
select timestamp as time, datetime(timestamp, 'unixepoch') AS utc_time, storage, message, actiity, process, pid, sender, tid, category, subsystem FROM unified_log WHERE (sender = 'AppleSystemPolicy' AND message like '%ASP: Security policy%');
seph
timestamp > -1
as a condition.lvferdi
10/19/2022, 1:09 PMDaniel Bretón Suárez
11/04/2022, 10:54 AMosquery.db
), it will reset the internal pointer of the evented style (timestamp > -1
).