👋 hi all i was taking another look at https://github.com/osquery/osquery/issues/7306 (getting accurate/full/large file listings) and trying to come up with a way to address that gap/limitation based on comments in the issue and other places i think a new table dedicated to get file listings would be best due to the intricacies of the

file

table and globbing? I started working on some ideas and i think a table like

file_list

and something like

path_start, path_limit, path_filter

could accomplish both listing files and working with any filters to limit results (if an analyst wants to): - path_start - Starting path to do a file listing (required=True) - path_limit - Number of subdirectories to descend (additional=True, default is 1) - path_filter - Regex to apply any optional filters (additional=True) Example below:

osquery> select * from file_list where path_start='/' and path_filter=".*\.php" and path_limit=-1 limit 10;
+---------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------+-----------------------------+----------+-----+-----+------+--------+-------+------------+------------+------------+------------+------------+------------+---------+---------+------------+------------+-------------+-----------+
| path                                                                                                                      | directory                                                                                         | filename                    | inode    | uid | gid | mode | device | size  | block_size | atime      | mtime      | ctime      | btime      | hard_links | symlink | type    | path_start | path_limit | path_filter | bsd_flags |
+---------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------+-----------------------------+----------+-----+-----+------+--------+-------+------------+------------+------------+------------+------------+------------+---------+---------+------------+------------+-------------+-----------+
| /Users/puffycid/Projects/osquery/build/libs/src/patched-source/thrift/src/contrib/fb303/php/FacebookBase.php              | /Users/puffycid/Projects/osquery/build/libs/src/patched-source/thrift/src/contrib/fb303/php       | FacebookBase.php            | 29001954 | 501 | 20  | 0644 | 0      | 1880  | 4096       | 1634951770 | 1624419491 | 1634951768 | 1624419491 | 1          | 0       | regular | /          | -1         | .*\.php     |           |
| /Users/puffycid/Projects/osquery/build/libs/src/patched-source/thrift/src/lib/php/lib/Base/TBase.php                      | /Users/puffycid/Projects/osquery/build/libs/src/patched-source/thrift/src/lib/php/lib/Base        | TBase.php                   | 29003115 | 501 | 20  | 0644 | 0      | 13412 | 4096       | 1634951773 | 1624419492 | 1634951768 | 1624419492 | 1          | 0       | regular | /          | -1         | .*\.php     |           |
| /Users/puffycid/Projects/osquery/build/libs/src/patched-source/thrift/src/lib/php/lib/ClassLoader/ThriftClassLoader.php   | /Users/puffycid/Projects/osquery/build/libs/src/patched-source/thrift/src/lib/php/lib/ClassLoader | ThriftClassLoader.php       | 29003096 | 501 | 20  | 0644 | 0      | 5951  | 4096       | 1634951773 | 1624419492 | 1634951768 | 1624419492 | 1          | 0       | regular | /          | -1         | .*\.php     |           |
| /Users/puffycid/Projects/osquery/build/libs/src/patched-source/thrift/src/lib/php/lib/Exception/TApplicationException.php | /Users/puffycid/Projects/osquery/build/libs/src/patched-source/thrift/src/lib/php/lib/Exception   | TApplicationException.php   | 29003103 | 501 | 20  | 0644 | 0      | 2414  | 4096       | 1634951773 | 1624419492 | 1634951768 | 1624419492 | 1          | 0       | regular | /          | -1         | .*\.php     |           |
| /Users/puffycid/Projects/osquery/build/libs/src/patched-source/thrift/src/lib/php/lib/Exception/TException.php            | /Users/puffycid/Projects/osquery/build/libs/src/patched-source/thrift/src/lib/php/lib/Exception   | TException.php              | 29003105 | 501 | 20  | 0644 | 0      | 13587 | 4096       | 1634951773 | 1624419492 | 1634951768 | 1624419492 | 1          | 0       | regular | /          | -1         | .*\.php     |           |
| /Users/puffycid/Projects/osquery/build/libs/src/patched-source/thrift/src/lib/php/lib/Exception/TProtocolException.php    | /Users/puffycid/Projects/osquery/build/libs/src/patched-source/thrift/src/lib/php/lib/Exception   | TProtocolException.php      | 29003106 | 501 | 20  | 0644 | 0      | 1455  | 4096       | 1634951773 | 1624419492 | 1634951768 | 1624419492 | 1          | 0       | regular | /          | -1         | .*\.php     |           |
| /Users/puffycid/Projects/osquery/build/libs/src/patched-source/thrift/src/lib/php/lib/Exception/TTransportException.php   | /Users/puffycid/Projects/osquery/build/libs/src/patched-source/thrift/src/lib/php/lib/Exception   | TTransportException.php     | 29003104 | 501 | 20  | 0644 | 0      | 1199  | 4096       | 1634951773 | 1624419492 | 1634951768 | 1624419492 | 1          | 0       | regular | /          | -1         | .*\.php     |           |
| /Users/puffycid/Projects/osquery/build/libs/src/patched-source/thrift/src/lib/php/lib/Factory/TBinaryProtocolFactory.php  | /Users/puffycid/Projects/osquery/build/libs/src/patched-source/thrift/src/lib/php/lib/Factory     | TBinaryProtocolFactory.php  | 29003108 | 501 | 20  | 0644 | 0      | 1384  | 4096       | 1634951773 | 1624419492 | 1634951768 | 1624419492 | 1          | 0       | regular | /          | -1         | .*\.php     |           |
| /Users/puffycid/Projects/osquery/build/libs/src/patched-source/thrift/src/lib/php/lib/Factory/TCompactProtocolFactory.php | /Users/puffycid/Projects/osquery/build/libs/src/patched-source/thrift/src/lib/php/lib/Factory     | TCompactProtocolFactory.php | 29003109 | 501 | 20  | 0644 | 0      | 1150  | 4096       | 1634951773 | 1624419492 | 1634951768 | 1624419492 | 1          | 0       | regular | /          | -1         | .*\.php     |           |
| /Users/puffycid/Projects/osquery/build/libs/src/patched-source/thrift/src/lib/php/lib/Factory/TJSONProtocolFactory.php    | /Users/puffycid/Projects/osquery/build/libs/src/patched-source/thrift/src/lib/php/lib/Factory     | TJSONProtocolFactory.php    | 29003111 | 501 | 20  | 0644 | 0      | 1138  | 4096       | 1634951773 | 1624419492 | 1634951768 | 1624419492 | 1          | 0       | regular | /          | -1         | .*\.php     |           |
+---------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------+-----------------------------+----------+-----+-----+------+--------+-------+------------+------------+------------+------------+------------+------------+---------+---------+------------+------------+-------------+-----------+

Some of the benefits of full file listings (or large file listings) are: - hash the full file system - look for specific files (ex: look for webshells like php, jsp, asp, aspx, etc. Though reading contents would be an additional requirement to be more effective) - if/when? https://github.com/osquery/osquery/pull/7160 is merged you could get PE/MACHO/ELF metadata for all binaries - scan all files (or specific files) with the

yara

table - look for data staging by getting all files above a specific size (or common staging archives like zip, 7z, tar, cab, etc) - more? i included this summary in the issue as well are others interested in this implementation? interested in others thoughts?