https://github.com/osquery/osquery logo
Join Slack
Powered by
Hey :osquery_intensifies: I'm working on an osque...
# fleet
c
Hey osquery intensifies I'm working on an osquery YARA scan intended to cover the entire filesystem using path LIKE "/%%". However, I'm observing that the recursion isn't behaving as expected, leading to incomplete results. Current Behavior: • When using path LIKE "/%%" (with either sigurl or sigfile for YARA rules), the query only returns matches found directly in the root directory (e.g., a file like /settings.php). • It fails to find known matching files located in various subdirectories (e.g., /app-data/config.php, /app-data/scripts/utils.php, /service-files/config.php, etc. – using generic examples here). • If I target these subdirectories directly in the query (e.g., path LIKE "/app-data/%%"), osquery does successfully find the matching files within those specific paths. • The /%% query (whether run via our FleetDM instance or directly in osqueryi connected to the daemon) completes very quickly, typically in under a second (around 0.5s), and returns only the root-level matches. Troubleshooting & Configuration: • The osqueryd agent is version 5.17.0 (from logs). • It's managed by "Orbit" launcher and FleetDM. • osqueryd is running with --verbose (passed by Orbit). • The /opt/orbit/osquery.flags file includes --logger_min_status=0 and --enable_watchdog_debug=true. • The orbit launcher itself is running with --debug. • We're monitoring logs via journalctl -u orbit.service -f and also directly from osqueryd's filesystem logs in /opt/orbit/osquery_log/osqueryd.INFO. • The query is not currently denylisted by Fleet. • Resource consumption (CPU/memory) for this quick /%% query is low, and the osquery watchdog doesn't seem to be intervening, nor are cgroup limits (MemoryLimit 300M, CPUQuota 35%) being hit during these short executions. Given that the /%% query is so fast and only finds root-level files, it strongly suggests osquery is stopping its filesystem traversal very early. However, the verbose logs (both journalctl and osqueryd.INFO) haven't yet shown explicit errors or warnings about why the traversal is stopping (e.g., no clear "skipping path due to symlink loop" messages immediately obvious for the /%% case). Any advice on how to further diagnose this premature stop in the /%% recursion, or suggestions for alternative approaches to achieve a more comprehensive filesystem YARA scan with osquery in a Fleet-managed environment, would be greatly appreciated. Thanks!
Copy code
## Query:

SELECT
    path,
    matches,
    count,
    sigurl,
    strings
FROM
    yara
WHERE
    path LIKE "/%%"
    AND sigurl IN (
        '<https://raw.githubusercontent.com/Yara-Rules/rules/master/webshells/WShell_THOR_Webshells.yar>',
        '<https://raw.githubusercontent.com/Yara-Rules/rules/master/webshells/WShell_APT_Laudanum.yar>'
    )
    AND count > 0;
m
Hi @Calil Khalil! I think you're on the right track with the troubleshooting you've done so far. At this point, I would recommend filing a bug report with osquery.
k
I'd like to add that the odds of performance issues and/or denylisting while scanning the entire disk is extremely high, and I'd generally recommend avoiding it.
I see that you've been looking at that, it just had to be said 🙂
j
I think this is a known issue in osquery. Check if https://github.com/osquery/osquery/issues/7291 looks the same.
a
@John Speno so you're saying you think it's an osquery limitation than yara scanning per se on windows as the limitation?
j
I'm saying there's an issue with the 
file
table, nothing else.
a
very cool, I suppose the yara piece uses the file table? makes sense I suppose
c
Thanks for all the feedback and insights so far! I've continued testing, but I still haven't been successful in getting a full disk scan to work with 
path LIKE "/%%".
The query continues to return only root-level matches and completes very quickly, missing known files in subdirectories.
f
this is just not something that osquery is really designed to do i don't think you can reasonably expect this to work. perhaps you would have more luck deploying the yara binary to your machines and use fleet to orchestrate the execution of a script that launches yara itself locally to scan all these locations, but imo that is also a bad idea. realistically i would limit the scope to common directories you may be worried about, or based on some spotlight metadata, use a subquery to generate a dynamic list of files to scan based on some criteria that is relevant to your environment.
c
Well, I guess I've tried enough. I'm going to give up. Thanks to everyone, regardless.
f
I did some research into osquery + yara last year, pathing is definitely a pain point but a mandatory tradeoff with performance. https://developer.squareup.com/blog/leveraging-linux-internals-to-supercharge-osquery-malware-detection/
c
Interesting article! Indeed, the issue of pathing with osquery + YARA is a known point, and this trade-off with performance is something to consider. In my case, performance hasn't been an issue, as I have watchdogs enabled and a dedicated slice in systemd to control resources. The strangest thing for me is that the YARA scan is not being alerted by either osquery or watchdogs. And they don't seem to be ‘dropping’ the scan; it seems to be more of a limitation in the way the osquery file table operates than a performance problem in itself. I'd love to be able to use YARA to recursively scan the system from time to time, but this seems to be precisely a limitation of the file table.
f
you may be able to trick it a bit by using a CTE to generate a list of directories and then recursively scan within each of them vs. starting at 
/
? but yeah this is an uphill battle afaik.
one of the takeaways from my article is to focus on the paths of running processes as it is highly likely that if malware exists, it would be within those paths vs. a static file sitting somewhere on disk.
file eventing could also be used, a loop could be setup similar to using inotify or something and only scanning net new paths/files day over day or hour by hour etc.
c
Thanks for the CTE suggestion! Really, to try to get around the file table limitation in more comprehensive recursive scans, as you said, is an ‘uphill battle’, but this approach may help to segment. I'll try it out.
j
Yeah, nice post, @FG - I think I tried using 
yara
on some paths in 
/proc
and it appeared that it wouldn't work on them (I forget the details now). Nice to know that it does work in this case.
8 Views
Open in Slack
PreviousNext
Powered by