https://github.com/osquery/osquery logo
Join Slack
Powered by
Hi everyone:wave: I am trying to add prefetch supp...
# core
p
Hi everyone👋 I am trying to add prefetch support to osquery (windows). Prefetch is a windows artifact that can show evidence of file execution. However, starting with windows 8 the prefetch file is compressed using LZxpress Huffman. Therefore before osquery can parse the file it must be decompressed. Currently there are three ways to decompress lzxpress Huffman: 1. Using the builtin windows functions: decompress (https://docs.microsoft.com/en-us/windows/win32/api/compressapi/nf-compressapi-decompress) or RtlDecompressBufferEx (https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/ntifs/nf-ntifs-rtldecompressbufferex) 2. Use a third party library to parse the prefetch file (libscca https://github.com/libyal/libscca) 3. Implement the decompression algorithm manually (go-prefetch https://github.com/Velocidex/go-prefetch/blob/master/lzxpress.go) Right now I am trying option 1 before 2 or 3. In order to decompress lzxpress Huffman osquery would need an additional windows library called Cabinet.{dll/lib} Right now I am having trouble adding the additional library to osquery. Ive added Cabinet.lib to flags.cmake (where other windows lib files listed) under cmake/flags.cmake, however it looks like the library is still not getting added to osquery during the build process?
Copy code
PS C:\Users\bob\Projects\osquery\build> radare2.exe .\osquery\RelWithDebInfo\osqueryd.exe
 -- Error: There's a missing space before the opening parenthesis '('
[0x1417b1910]> il
[Linked libraries]
shlwapi.dll
rpcrt4.dll
kernel32.dll
user32.dll
shell32.dll
ole32.dll
oleaut32.dll
advapi32.dll
ntdll.dll
ws2_32.dll
iphlpapi.dll
netapi32.dll
version.dll
wtsapi32.dll
secur32.dll
dbghelp.dll
dbgeng.dll
bcrypt.dll
crypt32.dll
wintrust.dll
setupapi.dll
userenv.dll
wevtapi.dll

23 libraries
sadly im not an expert in cmakelists☚ī¸, do I need to modify additional cmakelist files (or other files) in order for cabinet.dll/lib/header to be included in the final binary? thanks!
m
The change in 
flags.cmake
should get the linker to pick 
cabinet.lib
. Here is the output of build log command line output 
cabinet.lib
Copy code
LINK_FLAGS = /machine:X86 /debug /INCREMENTAL /subsystem:console  /SUBSYSTEM:CONSOLE /INCREMENTAL:NO ntdll.lib ole32.lib oleaut32.lib ws2_32.lib iphlpapi.lib netapi32.lib rpcrt4.lib shlwapi.lib version.lib wtsapi32.lib wbemuuid.lib secur32.lib taskschd.lib dbghelp.lib dbgeng.lib bcrypt.lib crypt32.lib wintrust.lib setupapi.lib advapi32.lib userenv.lib wevtapi.lib shell32.lib gdi32.lib mswsock.lib cabinet.lib /WHOLEARCHIVE:J:/osquery/build/osquery/sql/osquery_sql.lib /WHOLEARCHIVE:J:/osquery/build/osquery/database/osquery_database_ephemeral.lib /WHOLEARCHIVE:
👍 1
The headers will be picked automatically from the common sdk path. I just added some dummy function from msdn example that exercises these functions and it works fine. https://docs.microsoft.com/en-us/windows/win32/cmpapi/using-the-compression-api-in-buffer-mode
One thing i forgot to mention, depending on where you are going to add the code for these API. You might have to deal with 
WIN32_LEAN_AND_FLAG
and few other macros that get passed from top level. You might want to supress them to from the relevant project cmake file. e.g. Here is what i had to do in example extension. In this case ,i actually ended up not defining in the top level cmake but from the extension project.
Copy code
J:\osquery\external>gd CMakeLists.txt
diff --git a/external/CMakeLists.txt b/external/CMakeLists.txt
index 7b1c9de5..31c00ab4 100644
--- a/external/CMakeLists.txt
+++ b/external/CMakeLists.txt
@@ -18,10 +18,16 @@ set(EXTERNALS_DIR "${CMAKE_SOURCE_DIR}/external")

 add_library(external_options INTERFACE)
 target_compile_options(external_options INTERFACE -DOSQUERY_EXTERNAL)
+message("removing definition WIN32_LEAN_AND_MEAN")
+remove_definitions(-DWIN32_LEAN_AND_MEAN)
+remove_definitions(-DWIN32_WINNT)
+add_definitions(-D_WIN32_WINNT=_WIN32_WINNT_WIN8)
+add_definitions(-DNTDDI_VERSION=NTDDI_WIN8)
 target_link_libraries(external_options INTERFACE
   osquery_cxx_settings
   osquery_sdk_pluginsdk
   osquery_extensions_implthrift
+  cabinet.lib
 )
Copy code
Dump of file J:\osquery\build\external\examples\read_only_table\read_only_table_extension.ext.exe

File Type: EXECUTABLE IMAGE

  Section contains the following imports:

    Cabinet.dll
                8F0078 Import Address Table
                9D572C Import Name Table
                     0 time date stamp
                     0 Index of first forwarder reference

                      Ordinal    43
                      Ordinal    45
                      Ordinal    40
p
hey just wanted to follow up this helped alot and set me on the right path thanks!
👍đŸŊ 1
f
@puffycid I am super excited you are working on this! I was investigating prefetch files earlier today and was delighted when I saw your comment pop up in my search.
🎉 1
This will be a big step forward in terms of bringing greater parity to the 
programs
story on Windows, compared to what is available for macOS 
apps
p
Awesome! I hit some snags getting the decompression working But I'm hoping I can resolve/fix it over the weekend
f
🤞 Good luck!
@puffycid How goes fighting the good fight on the prefetch stuff? Hit anymore roadblocks?
p
Hey @fritz unfortunately ive hit some issues. So it turns out that it appears prefetch files cannot be decompressed with Cabinent.dll functions. Even though the files r compressed with lzxpress Huffman, the compression format/algorithm appears to be slightly different than the lzxpress Huffman algorithm that Cabinent.dll supports. đŸ˜ĸ This kind of makes sense because all the other public prefetch parsers released so far use the function RtlDecompressBufferEx (ntifs.h) and not decompress (compressapi.h). https://github.com/EricZimmerman/PECmd, https://gist.github.com/EricZimmerman/95be73f6cd04882e57e6, https://github.com/Velocidex/go-prefetch Unfortunately calling RtlDecompressBufferEx requires the Windows Driver Kit (WDK), and would require osquery to call driver function, which I believe osquery tries to avoid using? (Although the ntifs.h header is part of the NTDLL.lib which osquery does use. But the header file is part of the WDK and not the SDK) So the only other options to parse a prefetch file are(?): * include a third party library (ex: https://github.com/libyal/libscca) * implementing the decompression algorithm from scratch Imo including an additional dependency just to parse a single forensic artifact is kind of overkill/unnecessary? So implementing the decompression algorithm from is the only possibility? đŸ˜ĸ Luckily the authors of go-prefetch also implemented the lzxpress Huffman algorithm from scratch in Go (~300 lines) as a backup method in case RtlDecompressBufferEx has issues or if the user wants to decompress prefetch files on macos or linux. Ive translated the Go code to C++ (go-prefetch is Apache license so there shouldn't be any licensing issues?), but I'm not getting the same output đŸ˜ĸ So im currently debugging what is causing the issue, but it has been a slow/slightly painful process. I still plan on working on/finishing it, but im also working on other osquery features I would like to add I can upload what I have far to git if you would like to try to review/take a look? Or if you have any other ideas or suggestions about different ways to maybe parse prefetch files that would be great! I would love to be wrong about the Cabinent.dll decompression support or if there is better way of parsing prefetch files.
a
I think you can easily import the 
RtlDecompressBufferEx
function with LoadLibrary/GetProcAddress to avoid a hard dependency on the wdk
😮 1
exports from ntdll.dll
p
Interesting! I will check this out Thanks!
Just to follow up This method worked! and I'm able to decompressed the files Thanks again!đŸĻœ
f
This is awesome news! 😄 Thanks for the helpful pointer @alessandrogario!! and for all the hard work, @puffycid!
👍 1
p
@fritz just a fyi the PR for prefetch parsing/table has been opened https://github.com/osquery/osquery/pull/7076 In case u would like to track it parrotshuffle
f
@puffycid 🙌 🙏
That is awesome news! I will definitely be keeping 👀 on it and hopefully building and testing the PR locally to give some feedback this week!
@puffycid Just a heads up, with my local build of your branch I cannot get the 
prefetch
table to return results, it just keeps choking:
Copy code
PS C:\osquery\build\osquery\Debug> .\osqueryd.exe -S --verbose
I0428 15:39:55.243664 80768 init.cpp:342] osquery initialized [version=4.8.0-24-g69586fde5]
I0428 15:39:55.246650 80768 extensions.cpp:453] Could not autoload extensions: Cannot open file for reading: \Program Files\osquery\extensions.load
I0428 15:39:55.247650 80768 dispatcher.cpp:78] Adding new service: ExtensionWatcher (000001F234277990) to thread: 77380 (000001F23425C830) in process 73144
I0428 15:39:55.247650 80768 dispatcher.cpp:78] Adding new service: ExtensionRunnerCore (000001F234281DF0) to thread: 70328 (000001F23425BF70) in process 73144
I0428 15:39:55.248647 70328 interface.cpp:270] Extension manager service starting: \\.\pipe\shell.em
I0428 15:39:55.248647 80768 auto_constructed_tables.cpp:97] Removing stale ATC entries
Using a [1mvirtual database[0m. Need help, type '.help'

osquery> SELECT * FROM prefetch WHERE path = 'C:\Windows\Prefetch\<http://7ZG.EXE-F49B3D46.pf|7ZG.EXE-F49B3D46.pf>';
W0428 15:40:44.938586 80768 prefetch.cpp:253] Unsupported prefetch file: C:\Windows\Prefetch\<http://CHROME.EXE-AED7BA3D.pf|CHROME.EXE-AED7BA3D.pf>
W0428 15:40:45.093174 80768 prefetch.cpp:253] Unsupported prefetch file: C:\Windows\Prefetch\<http://CHROME.EXE-AED7BA44.pf|CHROME.EXE-AED7BA44.pf>
đŸ˜ĸ 1
I have tried using a 
WHERE filename
, a 
WHERE path
, it just always hangs on those Chrome prefetch files, even if I scope the query to not touch them.
p
Hey @fritz what windows system r u using? Can u share the chrome prefetch file? Is Zimmerman's prefetch parser able to parse it? (PEcmd) https://ericzimmerman.github.io/#!index.md
f
This is a Razer Blade Model# RZ09-01652:
Copy code
+-----------------+--------------------------+
| Column          | Value                    |
+-----------------+--------------------------+
| Build           | 19042                    |
| Codename        | Microsoft Windows 10 Pro |
| Display Version | 20H2                     |
| First Set Up On |                          |
| Major           | 10                       |
| Minor           | 0                        |
| Name            | Microsoft Windows 10 Pro |
| Patch           | (NULL)                   |
| Platform        | windows                  |
| Platform Like   | windows                  |
| Release ID      | 2009                     |
| UBR             | 928                      |
| Version         | 10.0.19042               |
+-----------------+--------------------------+
Let me check the parser against that file
and I will try to get you the prefetch file
So interestingly, it looks like these might be malformed in some way:
👍 1
Copy code
PS C:\Windows\Prefetch> C:\Users\kolide-razer\Downloads\Get-ZimmermanTools\PECmd.exe -f C:\Windows\Prefetch\<http://CHROME.EXE-AED7BA3D.pf|CHROME.EXE-AED7BA3D.pf>
PECmd version 1.4.0.0

Author: Eric Zimmerman (<mailto:saericzimmerman@gmail.com|saericzimmerman@gmail.com>)
<https://github.com/EricZimmerman/PECmd>

Command line: -f C:\Windows\Prefetch\<http://CHROME.EXE-AED7BA3D.pf|CHROME.EXE-AED7BA3D.pf>

Keywords: temp, tmp

Processing 'C:\Windows\Prefetch\<http://CHROME.EXE-AED7BA3D.pf|CHROME.EXE-AED7BA3D.pf>'

Error opening 'C:\Windows\Prefetch\<http://CHROME.EXE-AED7BA3D.pf|CHROME.EXE-AED7BA3D.pf>'. Message: Invalid signature! Should be 'SCCA'
When I run it through zimmerman tools it gets grumpy
69 Views
Open in Slack
PreviousNext
Powered by