https://github.com/osquery/osquery logo
Join Slack
Powered by
Has anyone had any luck with hidden files (like `/...
# macos
s
Has anyone had any luck with hidden files (like 
/Users/*/.zsh_history
) being tracked by the 
es_process_file_events
table?
s
hey Sean! That should work (unless of course I missed something and it’s a bug), what issues are you running into?
And what does your config look like?
s
Copy code
# Fleet configurations
--config_plugin=tls
--config_tls_endpoint=/api/v1/osquery/config
--config_tls_refresh=60
--distributed_interval=10
--distributed_plugin=tls
--distributed_tls_max_attempts=3
--distributed_tls_read_endpoint=/api/v1/osquery/distributed/read
--distributed_tls_write_endpoint=/api/v1/osquery/distributed/write
--events_expiry=1
--disable_distributed=false
--enroll_tls_endpoint=/api/v1/osquery/enroll
--tls_hostname=<server fqdn>
--tls_server_certs=/var/osquery/amazon.crt
--enroll_secret_path=/var/osquery/kolide_secret

# Logs results to Fleet server 
--logger_plugin=tls
--logger_tls_endpoint=/api/v1/osquery/log
--logger_tls_period=10

# When True, each row from a snapshot event is logged individually
--logger_snapshot_event_type=true

# Sets host identifier to hostname
--host_identifier=hostname

# Disables sensitive tables
--disable_tables=shell_history

# Sets query pack delimiter to "/"
--pack_delimiter=/

# Watchdog configs
--watchdog_memory_limit=500
--watchdog_utilization_limit=130

# Eventing configurations
--disable_events=false
--disable_endpointsecurity=false
--disable_endpointsecurity_fim=false
--es_fim_enable_open_events=true
--enable_keyboard_events=true
--enable_mouse_events=true

# Displays matching rule strings from yara scan results
--enable_yara_string=true
using Fleet, this is my file_paths config:
Copy code
overrides:
      platforms:
        darwin:
          file_paths:
            aws:
              - /Users/*/.aws/*
            ssh:
              - /Users/*/.ssh/*
            kubernetes:
              - /Users/*/.kube/config
            azure:
              - /Users/*/.azure/*.json
            chrome:
              - /Users/*/Library/Application\ Support/Google/Chrome/*/Bookmarks
              - /Users/*/Library/Application\ Support/Google/Chrome/*/Cookies
              - /Users/*/Library/Application\ Support/Google/Chrome/*/History
              - /Users/*/Library/Application\ Support/Google/Chrome/*/Login\ Data
            edge:
              - /Users/*/Library/Application\ Support/Microsoft/Edge/*/Bookmarks
              - /Users/*/Library/Application\ Support/Microsoft/Edge/*/Cookies
              - /Users/*/Library/Application\ Support/Microsoft/Edge/*/History
              - /Users/*/Library/Application\ Support/Microsoft/Edge/*/Login\ Data
            island:
              - /Users/*/Library/Application\ Support/Island/*/Bookmarks
              - /Users/*/Library/Application\ Support/Island/*/Cookies
              - /Users/*/Library/Application\ Support/Island/*/History
              - /Users/*/Library/Application\ Support/Island/*/Login\ Data
            gcloud:
              - /Users/*/.config/gcloud/*.db
              - /Users/*/.config/gcloud/*.json
              - /Users/*/.config/gcloud/legacy_credentials/*
              - /Users/*/.config/gcloud/cache/*
              - /Users/*/.config/gcloud/configurations/*
            teleport_certs:
              - /Users/%/.tsh/keys/%/cas/%.pem
            shell_history:
              - /Users/%/%_history
it's working great, except I'm not able to generate events for hidden files while other non-hidden files in monitored paths are working
Copy code
➜  ~ echo "shit" >> ~/.aws/.shit
➜  ~ cat ~/.aws/credentials
[default]
aws_access_key_id = <key_id>
aws_secret_access_key = <secret>
➜  ~ cat ~/.aws/.shit
shit
^ my crude example above generates an open event for the credentials file but not for the 
.shit
file
same goes for a dedicated entry for .bash_history and .zsh_history
s
That’s the feature I implemented — afaik there is nothing special about hidden files, so that should work, but let me get my infra up and running and I will check
s
Copy code
{ [-]
   action: added
   calendarTime: Tue Jan  9 17:53:21 2024 UTC
   columns: { [-]
     dest_filename:
     event_type: open
     filename: /Users/scavanaugh/.aws/credentials
     global_seq_num: 134337
     parent: 99895
     path: /bin/cat
     pid: 90364
     seq_num: 21666
     time: 1704822796
     version: 7
   }
   counter: 749
   epoch: 0
   hostIdentifier: <hostname>
   name: pack/macpack/es_file_events
   numerics: false
   unixTime: 1704822801
}
s
Can you try with absolute path? And is there anything in the verbose logs?
s
try which with the absolute path? update the config w/ the absolute path or the validation event?
s
Update the config with the absolute path (just trying to rule out if it’s globbing related or not), so for instance, instead of this:
Copy code
- /Users/*/.aws/credentials
Maybe try 
/Users/scavanaugh/.shit
in the config?
s
Will give that a go, one sec
Copy code
{ [-]
   action: added
   calendarTime: Tue Jan  9 18:23:27 2024 UTC
   columns: { [-]
     dest_filename:
     event_type: open
     filename: /Users/scavanaugh/.shit
     global_seq_num: 69
     parent: 99895
     path: /bin/cat
     pid: 97373
     seq_num: 4
     time: 1704824605
     version: 7
   }
   counter: 1042
   epoch: 0
   hostIdentifier: <hostname>
   name: pack/macpack/es_file_events
   numerics: false
   unixTime: 1704824607
}
explicit looks like its working
will update rule and wildcard username and retest
s
I have the same results, explicit is working, so I think it is a globbing bug, and potentially not just in the macOS tables
s
wildcarding username with 
*
worked
s
If you have access to verbose logs, it will show you which paths got picked up:
Copy code
I0109 23:55:11.791663 -4677376 endpointsecurity_fim.cpp:184] Monitoring path: /Users/sharvil/.aws/cli/
I0109 23:55:11.791697 -4677376 endpointsecurity_fim.cpp:184] Monitoring path: /Users/sharvil/.aws/config
I0109 23:55:11.791707 -4677376 endpointsecurity_fim.cpp:184] Monitoring path: /Users/sharvil/.aws/sso/
I0109 23:55:11.791714 -4677376 endpointsecurity_fim.cpp:184] Monitoring path: /Users/sharvil/.ssh/config
I0109 23:55:11.791723 -4677376 endpointsecurity_fim.cpp:184] Monitoring path: /Users/sharvil/.ssh/id_ed25519
I0109 23:55:11.791731 -4677376 endpointsecurity_fim.cpp:184] Monitoring path: /Users/sharvil/.ssh/id_ed25519.pub
I0109 23:55:11.791739 -4677376 endpointsecurity_fim.cpp:184] Monitoring path: /Users/sharvil/.ssh/known_hosts
I0109 23:55:11.791746 -4677376 endpointsecurity_fim.cpp:184] Monitoring path: /Users/sharvil/.ssh/known_hosts.old
I0109 23:55:11.791754 -4677376 endpointsecurity_fim.cpp:184] Monitoring path: /Users/sharvil/.ssh/.foo
I0109 23:55:11.791761 -4677376 endpointsecurity_fim.cpp:184] Monitoring path: /Library/Keychains/System.keychain
I0109 23:55:11.791769 -4677376 endpointsecurity_fim.cpp:184] Monitoring path: /Library/Keychains/apsd.keychain
Ah, that’s great, so I think one glob/wildcard would work, now I don’t remember if that’s a limitation or not, I will have to dive into the globbing code
s
any difference between 
*
and 
%
for wildcards under the hood?
s
I don’t think so
%
is SQL specific, but I think most folks are more used to 
*
, so we just support both (and it’s the same code path under the hood)
Please feel free to open an issue and tag me on it (I am happy to open too), and I will try and dig deeper into it
s
I retested explicit vs wildcard with a new file name and only the explicit path config returned results
I may have had a stale config w/ my last testing
I'll open an issue and save you some time
ty 1
s
I think the confusion resides in the globbing rules, since we use 
glob()
to get the entries. 
*
doesn't match hidden files, 
.*
does (or 
.%
).
but it's indeed annoying, and then I'm not sure if it also picks up the 
.
and 
..
directories
s
ah interesting, is there a reasonable path forward to include hidden files?
I'm confused why 
/Users/*/.foo
wouldn't work while 
/Users/scavanaugh/.foo
would work.
s
Not sure, but that seems a separate issue
s
It's essentially the root of this issue though, I was attempting to monitor the file_path value 
/Users/*/.zsh_history
and wasn't seeing validation events. Maybe the globbing isn't the culprit here?
s
Oh sorry, I think I misunderstood that your primary interest was with globbing on all hidden files. So the glob on all hidden files has the shortcoming I told above, where 
*
or 
%
doesn't really mean all files into the folder. This last issue (or your actual issue) seems different. It could still be related to something weird happening when parsing the path and the globs, but this one needs a closer look.
s
cool thanks for clarification @Stefano Bonicatti
17 Views
Open in Slack
PreviousNext
Powered by