Title
#general
puffycid

puffycid

08/28/2021, 3:07 AM
i have quick question does anyone have a query to get a full file listing of a system using osquery? i tried the one mentioned at https://blog.kolide.com/the-file-table-in-osquery-is-amazing-99db0f52a066 but it does not seem to be returning all the files (i may be doing something wrong) i also tried searching slack but did not get any hits thanks
s

seph

08/28/2021, 1:47 PM
does anyone have a query to get a full file listing of a system using osquery
Generally speaking this is a pretty hard thing. I don’t know how to do it, short of a really intense disk crawl. I think that’s true for all platform and software, really.
puffycid

puffycid

08/28/2021, 2:09 PM
darn thats kind of major limitation/gap from a forensic perspective is it a design decision or something that just hasnt been added because no one has volunteered to implement it? there are probably some technical challenges to implementing it, but generally speaking i think most forensic/live response software can pull a full file listing (i realize osquery is used for alot of other reasons/use cases, and a full file listing may not be as important for other use cases)
s

seph

08/29/2021, 12:29 AM
Technically speaking, I’m not aware of any tool that does this without crawling the filesystem. Sometimes in off hours. Usually at high performance cost.
12:29 AM
I’m not sure it’s the kind of thing we’d want directly supported. It feels a bit weird privacy wise, and I think an implementation wouldn’t fit very well.
puffycid

puffycid

08/29/2021, 1:22 AM
yes, getting a full file listing requires crawling the full file system? 😅 there are several forensic tools that can do this (in addition to hashing (md5) all files, and other stuff), this is a standard practice when doing a full IR investigation (vs triaging) for linux and mac a full disk crawl/file listing is often a requirement if a full live response investigation is being done for windows parsing the MFT file may be enough to get a full file listing (contains metadata for each file/directory on disk) can u clarify the privacy question/thought? its just metadata of all the files (ex: filename, directory, timestamps, attributes?) a full file listing would only be needed when doing an investigation, (maybe hunting too, with limits ex: all files that have the extension ".dmp" or all zip files greater than 10GB in size (potential data staging))
s

seph

08/31/2021, 2:19 AM
My gut sense is that there are two areas to work through. I don’t know the answer to either of them. this is a pretty big proposal.
2:20 AM
First. There’s the technical implementation. I don’t think this can be done well on-demand. There’s just too much data. I think to do this right would require something that crawled in the background and maintained state.Analogous to spotlight, or other security tools with this kind of functionality. It would be a big thing to add to osquery.
2:21 AM
Second. Privacy. I’m not sure where this falls.
2:22 AM
I’m hesitant to try to wrap my head around either of those separately, They’d both need to be answered.
2:22 AM
OTOH. if there are reasonable other tools that do this, is there an extension way to get at this?
puffycid

puffycid

08/31/2021, 5:54 AM
Understood, I may try playing around with some ideas I have Though I'm sure others have tried them 😄 This sounds like it could be a good office hours discussion? I can't attend this weeks office hours But the next one on sept 14? I should be able to attend \o.o/ When I reference other tools I was talking more along the lines of other EDR/IR/forensic applications No of which would work well as an extension I think? (Unless the extension shells out to run the application)
s

seph

08/31/2021, 3:16 PM
You could also open a ticket, and try to get conversation going.
3:16 PM
Ignoring the privacy side for a moment… I see 2 ways this could go.
3:16 PM
One, is adding robustness to the file table to allow an intense on-demand crawl.
3:17 PM
The other is building a subsystem to try doing this more asyncronously and more gently. (I think that pattern is common in EDR tools.)