Title
#fim
Schnoogemetzger

Schnoogemetzger

08/04/2021, 9:44 AM
Hi everyone 👋, I'm trying to use the FIM for monitoring log files under /var/log. Of course, due to the fact that these files are log files, they are updated and growing regularly. I would like to monitor only modifications of the log files, if the file size has not increased or it has shrunk (e.g. someone deleted some content). Is there a way with osquery to accomplish this? I did some tests and I saw the action "MOVED_TO" in the file_events table, if I'm going to delete some content of a file, but I don't know if this is enough. Maybe an approach would be, to compare the values of the size column, if this is possible 🤔
s

seph

08/04/2021, 4:06 PM
Hrm. Thinking aloud… FIM is generally good for tracking seldom changing files. Then you know you get the events.
4:07 PM
If these are generally growing, but you wanted to know when they shrunk, you might do better with a simple schedule query on the file table.
4:07 PM
That would return the size each time. But you’d need to alert on how that changed at some log aggregation level
4:08 PM
I wonder what circumstances would cause an inode change. Or if there are specific audit style events you could look fort
v

vaar

08/10/2021, 9:35 PM
logrotate maybe?
Schnoogemetzger

Schnoogemetzger

09/01/2021, 1:02 PM
Thank you. After some testing with different selects, this works with osqueryi, but not as a query for a scheduled file events in the osquery.conf
SELECT * FROM file AS foo WHERE foo.path like '/home/user/logtester/%%' AND foo.mtime >= (SELECT strftime('%s','now')-120) AND EXISTS ( SELECT * FROM file_events AS bar WHERE bar.action in ('MOVED_TO','DELETED','ATTRIBUTES_MODIFIED','UPDATED') AND foo.path = bar.target_path AND foo.mtime > bar.mtime AND foo.size < bar.size );
If this query is configured I get the following error:
Error adding new results to database for query file_events: JSON object was not an array
If I'm going to change the query a little bit and ensure that the file_events will be queried first, than there are no errors:
SELECT * FROM file_events AS bar WHERE bar.action in ('MOVED_TO','DELETED','ATTRIBUTES_MODIFIED','UPDATED') AND EXISTS ( SELECT * FROM file AS foo WHERE foo.path like '/home/user/logtester/%%' AND foo.path = bar.target_path AND foo.mtime > bar.mtime AND foo.size < bar.size AND foo.mtime >= (SELECT strftime('%s','now')-120));
I could live with this solution, but it doesn't work with scheduling. I can see with the daemon verbose mode, that the query will be executed, but the results will not be logged. If I'm going to do the same select with osqueryi I got my results as expected. So I'm a little bit clueless right now and I ask myself: Is it even possible to monitor decreasing files with the FIM functionality of osquery? We need this if, for example, someone deletes lines from log files. Do you have any further idea how to do this?
zwass

zwass

09/07/2021, 2:10 PM
Error adding new results to database for query file_events: JSON object was not an array
-- This looks like a bug that should be addressed. Can you please file an issue on GitHub?
Schnoogemetzger

Schnoogemetzger

09/07/2021, 2:18 PM
Sure 🙂
2:19 PM
What is your genaral opinion about it? Is osquery the right tool to monitor decreasing logs?
zwass

zwass

09/07/2021, 2:57 PM
Interesting question. This seems potentially tricky as there may be edge cases such as multiple processes writing the same file, simultaneous additions/deletions, etc. and of course you won't get the diff of what's been "removed" from the file. Any tool that does such a thing comprehensively will be very expensive (in terms of resource utilization) to operate on busy files like log files.
3:00 PM
If this is your primary use case I'd lean towards "osquery is not the right tool", but if you've already got (or will add) osquery and want some visibility into this, it may be reasonable to do.
Schnoogemetzger

Schnoogemetzger

09/09/2021, 7:03 AM
Hi @zwass Thank you for your answer. The reason behind this is the following: Your logs will be written resp. forwarded to a log server. At this log server only one process is going to write these logs (rsyslogd in our case). So, this is not the problem at all. In a normal situtation, the log file will never shrink, but what happens if someone (e.g. an attacker) is going to change these log files and delete, let's say, 20 lines of it? Sure, the lines are gone, but I want to know that the log file shrunk, because there is something going on. There are IDS / FIM tools which can handle such things, but osquery obviously not. The big question for me: Is this achievable with osquery without ten thousend of SELECT workarounds? 😅
s

seph

09/17/2021, 3:57 PM
Thinking aloud.. You’re asking how to tell whether an attackers has modified a file before the contents have been shipped off disk. Correct?
3:58 PM
I don’t think you can do this with file size. First, if you never rotate things, they will eventually consume all disk. Second, if these files are growing, deleting a line won’t really be noticeable as a file change.
3:58 PM
I think you could probably look at the audit systems, and see if you can identify what kinds of audit events are triggerable, vs what is ignorable.
3:59 PM
Or you could look at shipping the logs elsewhere, and other certified logging systems