Hi everyone wave I am trying to add prefetch support to osqu

Question

Hi everyone wave I am trying to add prefetch support to osquery windows Prefetch is a windows artifact that can show evidence of file execution However starting with windows 8 the prefetch file is compressed using LZxpress Huffman Therefore before osquery can parse the file it must be decompressed Currently there are three ways to decompress lzxpress Huffman 1 Using the builtin windows functions decompress or RtlDecompressBufferEx 2 Use a third party library to parse the prefetch file libscca 3 Implement the decompression algorithm manually go prefetch Right now I am trying option 1 before 2 or 3 In order to decompress lzxpress Huffman osquery would need an additional windows library called Cabinet dll lib Right now I am having trouble adding the additional library to osquery Ive added Cabinet lib to flags cmake where other windows lib files listed under cmake flags cmake however it looks like the library is still not getting added to osquery during the build process ```PS C Users bob Projects osquery build gt radare2 exe osquery RelWithDebInfo osqueryd exe Error There s a missing space before the opening parenthesis 0x1417b1910 gt il Linked libraries shlwapi dll rpcrt4 dll kernel32 dll user32 dll shell32 dll ole32 dll oleaut32 dll advapi32 dll ntdll dll ws2 32 dll iphlpapi dll netapi32 dll version dll wtsapi32 dll secur32 dll dbghelp dll dbgeng dll bcrypt dll crypt32 dll wintrust dll setupapi dll userenv dll wevtapi dll 23 libraries``` sadly im not an expert in cmakelists white frowning face do I need to modify additional cmakelist files or other files in order for cabinet dll lib header to be included in the final binary thanks

manu · Answer

The change in `flags cmake` should get the linker to pick `cabinet lib` Here is the output of build log command line output `cabinet lib` ```LINK FLAGS = machine X86 debug INCREMENTAL subsystem console SUBSYSTEM CONSOLE INCREMENTAL NO ntdll lib ole32 lib oleaut32 lib ws2 32 lib iphlpapi lib netapi32 lib rpcrt4 lib shlwapi lib version lib wtsapi32 lib wbemuuid lib secur32 lib taskschd lib dbghelp lib dbgeng lib bcrypt lib crypt32 lib wintrust lib setupapi lib advapi32 lib userenv lib wevtapi lib shell32 lib gdi32 lib mswsock lib cabinet lib WHOLEARCHIVE J osquery build osquery sql osquery sql lib WHOLEARCHIVE J osquery build osquery database osquery database ephemeral lib WHOLEARCHIVE ```

manu · Answer

The headers will be picked automatically from the common sdk path I just added some dummy function from msdn example that exercises these functions and it works fine <https docs microsoft com en us windows win32 cmpapi using the compression api in buffer mode>

manu · Answer

One thing i forgot to mention depending on where you are going to add the code for these API You might have to deal with `WIN32 LEAN AND FLAG` and few other macros that get passed from top level You might want to supress them to from the relevant project cmake file e g Here is what i had to do in example extension In this case i actually ended up not defining in the top level cmake but from the extension project ```J osquery external gt gd CMakeLists txt diff git a external CMakeLists txt b external CMakeLists txt index 7b1c9de5 31c00ab4 100644 a external CMakeLists txt +++ b external CMakeLists txt 18 10 +18 16 set EXTERNALS DIR $ CMAKE SOURCE DIR external add library external options INTERFACE target compile options external options INTERFACE DOSQUERY EXTERNAL +message removing definition WIN32 LEAN AND MEAN +remove definitions DWIN32 LEAN AND MEAN +remove definitions DWIN32 WINNT +add definitions D WIN32 WINNT= WIN32 WINNT WIN8 +add definitions DNTDDI VERSION=NTDDI WIN8 target link libraries external options INTERFACE osquery cxx settings osquery sdk pluginsdk osquery extensions implthrift + cabinet lib ```

manu · Answer

```Dump of file J osquery build external examples read only table read only table extension ext exe File Type EXECUTABLE IMAGE Section contains the following imports Cabinet dll 8F0078 Import Address Table 9D572C Import Name Table 0 time date stamp 0 Index of first forwarder reference Ordinal 43 Ordinal 45 Ordinal 40```

puffycid · Answer

hey just wanted to follow up this helped alot and set me on the right path thanks

fritz · Answer

< puffycid> I am super excited you are working on this I was investigating prefetch files earlier today and was delighted when I saw your comment pop up in my search

fritz · Answer

This will be a big step forward in terms of bringing greater parity to the `programs` story on Windows compared to what is available for macOS `apps`

puffycid · Answer

Awesome I hit some snags getting the decompression working But I m hoping I can resolve fix it over the weekend

fritz · Answer

crossed fingers Good luck

fritz · Answer

< puffycid> How goes fighting the good fight on the prefetch stuff Hit anymore roadblocks

puffycid · Answer

Hey < fritz> unfortunately ive hit some issues So it turns out that it appears prefetch files cannot be decompressed with Cabinent dll functions Even though the files r compressed with lzxpress Huffman the compression format algorithm appears to be slightly different than the lzxpress Huffman algorithm that Cabinent dll supports cry This kind of makes sense because all the other public prefetch parsers released so far use the function RtlDecompressBufferEx ntifs h and not decompress compressapi h Unfortunately calling RtlDecompressBufferEx requires the Windows Driver Kit WDK and would require osquery to call driver function which I believe osquery tries to avoid using Although the ntifs h header is part of the NTDLL lib which osquery does use But the header file is part of the WDK and not the SDK So the only other options to parse a prefetch file are include a third party library ex implementing the decompression algorithm from scratch Imo including an additional dependency just to parse a single forensic artifact is kind of overkill unnecessary So implementing the decompression algorithm from is the only possibility cry Luckily the authors of go prefetch also implemented the lzxpress Huffman algorithm from scratch in Go ~300 lines as a backup method in case RtlDecompressBufferEx has issues or if the user wants to decompress prefetch files on macos or linux Ive translated the Go code to C++ go prefetch is Apache license so there shouldn t be any licensing issues but I m not getting the same output cry So im currently debugging what is causing the issue but it has been a slow slightly painful process I still plan on working on finishing it but im also working on other osquery features I would like to add I can upload what I have far to git if you would like to try to review take a look Or if you have any other ideas or suggestions about different ways to maybe parse prefetch files that would be great I would love to be wrong about the Cabinent dll decompression support or if there is better way of parsing prefetch files

alessandrogario · Answer

I think you can easily import the `RtlDecompressBufferEx` function with LoadLibrary GetProcAddress to avoid a hard dependency on the wdk

alessandrogario · Answer

exports from ntdll dll

puffycid · Answer

Interesting I will check this out Thanks

puffycid · Answer

Just to follow up This method worked and I m able to decompressed the files Thanks again partyparrot

fritz · Answer

This is awesome news smile Thanks for the helpful pointer < alessandrogario> and for all the hard work < puffycid>

puffycid · Answer

< fritz> just a fyi the PR for prefetch parsing table has been opened In case u would like to track it parrotshuffle

fritz · Answer

< puffycid> raised hands pray

fritz · Answer

That is awesome news I will definitely be keeping eyes on it and hopefully building and testing the PR locally to give some feedback this week

fritz · Answer

< puffycid> Just a heads up with my local build of your branch I cannot get the `prefetch` table to return results it just keeps choking ```PS C osquery build osquery Debug gt osqueryd exe S verbose I0428 15 39 55 243664 80768 init cpp 342 osquery initialized version=4 8 0 24 g69586fde5 I0428 15 39 55 246650 80768 extensions cpp 453 Could not autoload extensions Cannot open file for reading Program Files osquery extensions load I0428 15 39 55 247650 80768 dispatcher cpp 78 Adding new service ExtensionWatcher 000001F234277990 to thread 77380 000001F23425C830 in process 73144 I0428 15 39 55 247650 80768 dispatcher cpp 78 Adding new service ExtensionRunnerCore 000001F234281DF0 to thread 70328 000001F23425BF70 in process 73144 I0428 15 39 55 248647 70328 interface cpp 270 Extension manager service starting pipe shell em I0428 15 39 55 248647 80768 auto constructed tables cpp 97 Removing stale ATC entries Using a 1mvirtual database 0m Need help type help osquery gt SELECT FROM prefetch WHERE path = C Windows Prefetch W0428 15 40 44 938586 80768 prefetch cpp 253 Unsupported prefetch file C Windows Prefetch W0428 15 40 45 093174 80768 prefetch cpp 253 Unsupported prefetch file C Windows Prefetch ```

fritz · Answer

I have tried using a `WHERE filename` a `WHERE path` it just always hangs on those Chrome prefetch files even if I scope the query to not touch them

puffycid · Answer

Hey < fritz> what windows system r u using Can u share the chrome prefetch file Is Zimmerman s prefetch parser able to parse it PEcmd

fritz · Answer

fritz · Answer

Let me check the parser against that file

fritz · Answer

and I will try to get you the prefetch file

fritz · Answer

So interestingly it looks like these might be malformed in some way

fritz · Answer

```PS C Windows Prefetch gt C Users kolide razer Downloads Get ZimmermanTools PECmd exe f C Windows Prefetch PECmd version 1 4 0 0 Author Eric Zimmerman Command line f C Windows Prefetch Keywords temp tmp Processing C Windows Prefetch Error opening C Windows Prefetch Message Invalid signature Should be SCCA ```

fritz · Answer

When I run it through zimmerman tools it gets grumpy