I should clarify that I thought that ` ` would recurse down

Question

I should clarify that I thought that ` ` would recurse down from the root directory but the top query is only return 3 results for some reason which is not what I thought it would do

mikermcneil · Answer

Hey Zander this isn t exactly an answer but in the mean time this made me think this is something that might be valuable in Fleet if so feel free to add context <https github com fleetdm fleet issues 1862>

mikermcneil · Answer

FWIW I get 20 results from both queries

mikermcneil · Answer

I see it going recursively deep

puffycid · Answer

I also thought ` ` would recursively go through the full file system but it usually went only 2 3 directories but I may have been doing something wrong as it looks like mikermcneil got some accurate results Also just to addon to the fleet issue feature idea In addition to listing recently modified files listing all created files in a specified time range could be very powerful valuable Ex scenario attacker logs onto system via RDP session lasts 4 hours Run query to list all files created during that 4 hour period Same idea with modified accessed etc

mikermcneil · Answer

Actually I bumped up my `LIMIT` to 200 to verify and I m seeing what looks like a max depth of 3 directories deep too

mikermcneil · Answer

< puffycid> got it thanks I ll add that feedback to the issue

mikermcneil · Answer

This behavior is unexpected to me too if intentional I think it deserves to be mentioned in the table documentation <https osquery io schema 4 9 0 file> I ll make a note to discuss at office hours if no one comes up with a solution before then

seph · Answer

Generally speaking this kind of thing is going to crawl disk And probably not work well

seph · Answer

Doing this well would require something with file events I don t know if you could do it wholly inside osquery or if you need the server to coordinate

mikermcneil · Answer

Seph do you know if the cap on depth is intentional

seph · Answer

I don t I d have to look at the source code Offhand it seems like a reasonable limit for resource utilization Crawling disk is a common way to lock machines

seph · Answer

I guess we have other limits exposed in various ways This could be too But I ve never seen a fa crawl be a good idea So I think it d be better to pursue this a different way

puffycid · Answer

Just to add my 2 cents about some of the reasons u may want a full file listing or at least scan the whole disk for a single system Look for all files created in a specified time range Count number of files encrypted by ransomware Hunt for webshells would also require reading regex file contents to be effective whole different discussion Data staging for exfil ex look for all files greater than 10GB Scan for indicators of compromise ex does the filename mimkatz exe exist anywhere on disk Some of the examples r applicable to one system others would be multiple systems Ex query all servers for websells all files with the extension php Vs Ex query all files created during the rdp session on serverA If others have other ideas and thoughts on how to get this data without a full file listing I would be very interested grinning

puffycid · Answer

Also just to add the stuff above is stuff I do at work So pulling a full file listing is definitely a common normal thing to do for incident reaponse IR forensics And since osquery can be used for IR forensics along with other stuff It would be nice to have this capability if an implementation was made that works well IMO

Zander Mackie · Answer

Agreed that this would be useful In my case I was doing IR on machine specifically walled off and snapshotted for this purpose so the resource use wasn t an issue In any case It would be nice to be able to as a user make that determination myself by tuning some value What I m trying to do is improve our IR processes and put them as much in osquery as possible To perform this task another way I d use `find` fully knowing it could lock the machine ```find src type f print0 | xargs 0 stat format Y y n | sort nr | cut d f2 | head n 20```

mikermcneil · Answer

Discussing in office hours <https osquery slack com archives C6PNW4528 p1630428308001500>

mikermcneil · Answer

goal is there a bug here if not can we document the limitation it in the table docs on Notes could be due to 50MB filesystem read limit that is configurable could try configuring that to rule out it s not this another potential culprit Stefano on linux we use the glob function not sure if it s necessarily the fastest way to crawl the fs On windows it is slower If interested in crawling the whole filesystem it s bound to be slow especially on Windows If we are interested in solving this we might consider giving a second look to Windows to improve performance Seph Could be another type of limit we aren t thinking of right now as well mikermcneil I ll make a bug issue < Zander Mackie> < puffycid> In the meantime it s potentially worth experimenting with the config for the filesystem read limit to see if it fixes this if so that could prove that the filesystem read limit is the problem See recording of office hours for more context

mikermcneil · Answer

Here s that issue <https github com osquery osquery issues 7291>

Zander Mackie · Answer

Left a comment to this effect in the issue but this does not appear to be fixed by adjusting `read max`

Zander Mackie · Answer

at least testing on macOS

seph · Answer

`read max` should be about max file size

Zander Mackie · Answer

gt could be due to 50MB filesystem read limit that is configurable could try configuring that to rule out it s not this Was this referring to `read max` or something else I was a bit confused because the docs agreed with what < seph> pointed out

sharvil · Answer

I wonder if osquery has full disk access permissions

seph · Answer

I thnk the read max was a speculation

Zander Mackie · Answer

Ah gotcha