https://github.com/osquery/osquery logo
Join Slack
Powered by
Hey there :wave: I’m seeing some strange behaviou...
# fim
p
Hey there 👋 I’m seeing some strange behaviour with inotify based FIM, which seems a little odd. Looking at the documentation, the wildcard 
%%
is used to denote that all files and folders should be matched, and thus monitored, recursively. Based on this, I should be able to recursively monitor for changes to CA certificates using 
/usr/share/ca-certificates/%%
? However, when this is done, osquery is attempting to install watches on all files under the specified directory, and subdirectories, as if they were themselves directories - which fails:
Copy code
W1215 15:29:47.814217 14737 inotify.cpp:371] Could not add inotify watch on: /usr/share/ca-certificates/mozilla/E-Tugra_Certification_Authority.crt/

W1215 15:29:47.814276 14737 inotify.cpp:371] Could not add inotify watch on: /usr/share/ca-certificates/mozilla/QuoVadis_Root_CA_3_G3.crt/

W1215 15:29:47.814344 14737 inotify.cpp:371] Could not add inotify watch on: /usr/share/ca-certificates/mozilla/ACCVRAIZ1.crt/
There are most certainly files under this path at the time that osquery was launched:
Copy code
root@ip-100-66-32-61:/tmp/inotify-info# find /usr/share/ca-certificates/mozilla/ -type f | wc -l
136
…but osquery isn’t installing watches on the files, just the directories.
Copy code
root@ip-100-66-32-61:/tmp/inotify-info# ./_release/inotify-info 15460 | grep /usr/share/ca-certificates/
    31382 [259:1] /usr/share/ca-certificates/
    31383 [259:1] /usr/share/ca-certificates/mozilla/
It looks like I should be within the currently configured Kernel limits, so I don’t believe this is the issue - although I may be wrong:
Copy code
root@ip-100-66-32-61:/tmp/inotify-info# lsof | grep inotify | wc -l
134

root@ip-100-66-32-61:/tmp/inotify-info# sysctl -a | grep -i max_user_watches
fs.inotify.max_user_watches = 29900
It looks like this code-path is being hit for children in subdirectories, which would explain the errors having a trailing 
/
on the file paths as if they were directories, but a watch is seemingly never installed on the files themselves.
Ahh. That’ll do it. When watching a directory inotify returns events for the directory and files inside of the directory. I’ll dig into where my events are wandering off to, but it looks like this isn’t the cause. Sorry for the noise!
c
Thanks for posting a follow up on the solution 🙂
r
<https://github.com/osquery/osquery/blob/master/osquery/events/linux/inotify.cpp#L402>
Copy code
addMonitor(canonicalized,isc,mask,false);
is setting 
recursive
to 
false
in the call to 
addMonitor
(leaving out param 
add_watch
) regardless of the value of 
recursive
as it was passed in, but I have no idea what that actually means
d
p
Looking at the code, it appears that the loop which iterates over children in a given sub-directory does not perform a subsequent check to see whether the child is a directory or not. As a result, all children have the 
/
appended and are attempted to be watched even if they’re files. This seems to be where this warning is originating, as files within these directories will fail to have a watch installed due to this appended 
/
.
In terms of the filtering of symlinks, I couldn’t work out a way to prevent the watches from being installed, but instead ended up using 
exclude_paths
to filter these for each category - where possible:
Copy code
"exclude_paths": {
        "configuration": [
            "/tmp/%%",
            "/dev/%%"
        ],
It does mean that there are wasted inotify handles due to these followed symlinks, but it does prevent these entries from being output at query time - reducing noise from our telemetry.
d
Personally I’ve started using 
file_paths_query
a bunch, however I start to wonder how computationally expensive that is. I also feel like having this being able to query for the 
directory
column rather than 
path
would be better, as placing an inotify watch there rather than files is preferable (as I understand it). Something like a 
"SELECT DISTINCT directory FROM file"
. Perhaps soon I’ll start logging issues and PRs 😄
f
it appears people have noticed this behavior in the past and filed these two issues: • https://github.com/osquery/osquery/issues/8240 • https://github.com/osquery/osquery/issues/8354 but this slack thread is the most detailed analysis of the problem that I have come across. thank you! we are seeing the same problem with the trailing slashes appended, accidentally treating files like directories, etc. might give 
file_paths_query
a try as a better(?) / different approach, but it would be nice if this issue was fixed in the code as well.
I start to wonder how computationally expensive that is
Do we know how often the 
file_paths_query
queries are executed to refresh the list of files/dirs that inotify events are placed on?
s
We reported the FIM bug to fleetdm and they pushed: https://github.com/osquery/osquery/pull/8399 which was released in 5.13.1. FWIW we encountered 
file_paths_query
hitting watchdog limits in certain situations (like a user who had extraced the linux kernel in their home directory)...
p
Huh neat! Thanks for doing that!
s
no problem! give it a try. in our case like 99% of the problem was related to osquery being unable to properly follow symlinks
3 Views
Open in Slack
PreviousNext
Powered by