https://github.com/osquery/osquery logo
Join Slack
Powered by
Hey all, so i'm still unable to get any ntfs_journ...
# fleet
t
Hey all, so i'm still unable to get any ntfs_journal_events data from my windows hosts, after recently adding file paths to my agent settings. I do however see data from other queries such as hardware events and firmware snapshots. My file events query:
Copy code
SELECT action, category, old_path, path, file_attributes, time FROM ntfs_journal_events;
my agent config
Copy code
config:
  options:
    events_expiry: 60
    config_refresh: 600
    host_identifier: instance
    distributed_interval: 60
  decorators:
    load:
      - >-
        SELECT COALESCE((select instance_id FROM ec2_instance_metadata),
        hostname) as hostname FROM system_info;
  file_paths:
    etc:
      - /etc/group
      - /etc/passwd
      - /etc/shadow
      - /etc/services
      - /etc/sudoers
      - /etc/ld.so.preload
      - /etc/ld.so.conf
      - /etc/ld.so.conf.d/%%
      - /etc/pam.d/%%
      - /etc/resolv.conf
      - /etc/modules
      - /etc/hosts
      - /etc/hostname
      - /etc/fstab
      - /etc/rsyslog.conf
    ssh:
      - /root/.ssh/%%
      - /home/%/.ssh/%%
      - /etc/ssh/%%
      - /var/lib/sia/keys/
      - /var/lib/sia/certs/
    logs:
      - /var/log/secure
    docker:
      - /etc/docker/%%
      - /etc/default/docker
      - /etc/docker/daemon.json
      - /usr/bin/containerd
      - /usr/sbin/runc
      - /etc/sysconfig/docker
      - /usr/lib/systemd/system/docker.service
      - /usr/lib/systemd/system/docker.socket
    osquery:
      - /etc/osquery/%%
      - /usr/share/osquery/packs/%%
    firewalls:
      - /etc/sysconfig/iptables
      - /home/y/conf/yakl/%%
      - /etc/yakl/conf/%%
overrides:
  platforms:
    windows:
      options:
        events_expiry: 60
        config_refresh: 600
        host_identifier: instance
        distributed_interval: 60
      decorators:
        load:
          - >-
            SELECT COALESCE((select instance_id FROM ec2_instance_metadata),
            hostname) as hostname FROM system_info;
      file_paths:
        users:
          - 'C:\users\AppData\Roaming'
          - 'C:\users\AppData\Local'
          - 'C:\users\AppData\Local\temp'
          - >-
            C:\users\AppData\Roaming\Microsoft\Windows\Start
            Menu\Programs\Startup
          - 'C:\users\AppData\Roaming\Microsoft\Windows\Start Menu\Programs'
          - 'C:\Users\Default'
        windows:
          - 'C:\Windows'
          - 'C:\Windows\temp'
          - 'C:\Windows\system32\Drivers'
          - 'C:\Windows\SysWOW64\Drivers'
          - 'C:\Windows\system32\GroupPolicy\Machine\Scripts'
          - 'C:\Windows\system32\GroupPolicy\User\Scripts'
          - 'C:\Windows\system32\Wbem'
          - 'C:\Windows\SysWOW64\Wbem'
          - 'C:\Windows\system32\WindowsPowerShell'
          - 'C:\Windows\SysWOW64\WindowsPowerShell'
          - 'C:\Windows\Tasks'
          - 'C:\Windows\system32\Tasks'
          - 'C:\Windows\AppPatch\Custom%'
        ProgramData:
          - 'C:\ProgramData\Microsoft\Windows\Start Menu'
          - 'C:\ProgramData\Microsoft\Windows\Start Menu\Programs'
      exclude_paths:
        windows:
          - 'C:\Windows\system32\DriverStore\Temp\%'
          - 'C:\Windows\system32\wbem\Performance%'
          - 'C:\$WINDOWS.~BT\Sources\%'
          - 'C:\Windows\Installer\%'
          - 'C:\Windows\System32\Tasks\Adobe Acrobat Update Task%'
          - 'C:\Windows\System32\Tasks\Adobe Flash Player Updater%'
          - >-
            C:\Windows\System32\Tasks\OfficeSoftwareProtectionPlatform\SvcRestartTask%
@Mystery Incorporated helped with this before, if you could just give me another look that would be much appreciated. I'm really lost.
m
@Ted Dorosheff show flags file please, what you need to set is probably in flags file
z
@Ted Dorosheff Try live querying 
select * from osquery_events
to see whether the publishers are activated and receiving any events.
t
hey @zwass happy new year. Back at the keyboard today, and i think you may be right about the event publishers not being activated.
Copy code
Log file created at: 2021/12/27 11:01:32
Running on machine: COOLEDRULED
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I1227 11:01:32.819942  5964 eventfactory.cpp:156] Event publisher not enabled: ntfs_event_publisher: NTFS event publisher disabled via configuration
z
Seems like @Mystery Incorporated has it right then -- you'll need to configure the appropriate flags.
👍 1
t
okay, do these look like the right flags?
Copy code
--enable_windows_events_publisher=true 
--enable_windows_events_subscriber=true 
--enable_ntfs_event_publisher=true
They were currently missing from my flags file
although going off the log, looks like i may only need
Copy code
--enable_ntfs_event_publisher=true
z
That sounds right. The 
windows_events
ones would be for the 
windows_events
table.
t
alright so i just tried live querying the host, and now at least i'm not getting the error about the event publisher not enabled, but i'm still not seeing any journal events
I ran osqueryi from cmd, with all those flags set.
Copy code
osqueryi.exe -S --config_path="C:\Program Files\osquery\osquery.conf" --disable_events=false --enable_windows_events_publisher=true --enable_windows_events_subscriber=true --enable_ntfs_event_publisher=true --verbose
But i'm guessing that file_paths still needs to be defined? We have file_paths set in our yaml config in fleetDM, but i would assume that isn't being loaded if i'm running osqueryi like this?
Copy code
osquery> select * from osquery_events;
+--------------------------+--------------------------+------------+---------------+--------+-----------+--------+
| name                     | publisher                | type       | subscriptions | events | refreshes | active |
+--------------------------+--------------------------+------------+---------------+--------+-----------+--------+
| WindowsEventLogPublisher | WindowsEventLogPublisher | publisher  | 1             | 17     | 0         | 1      |
| ntfs_event_publisher     | ntfs_event_publisher     | publisher  | 0             | 0      | 934       | 1      |
| ntfs_journal_events      | ntfs_event_publisher     | subscriber | 0             | 0      | 0         | 1      |
| powershell_events        | WindowsEventLogPublisher | subscriber | 0             | 0      | 0         | 0      |
| windows_events           | WindowsEventLogPublisher | subscriber | 1             | 437    | 0         | 1      |
+--------------------------+--------------------------+------------+---------------+--------+-----------+--------+
osquery>
z
Yeah that's right.
t
before i ran the query, i dropped a .txt file into one of the paths which we've got listed in our yaml config
z
Can you run 
osqueryi
with the normal flagfile so it pulls the config from Fleet?
t
yes
Copy code
C:\Program Files\osquery>osqueryi.exe
Using a [1mvirtual database[0m. Need help, type '.help'
osquery> select * from osquery_events;
+--------------------------+--------------------------+------------+---------------+--------+-----------+--------+
| name                     | publisher                | type       | subscriptions | events | refreshes | active |
+--------------------------+--------------------------+------------+---------------+--------+-----------+--------+
| WindowsEventLogPublisher | WindowsEventLogPublisher | publisher  | 0             | 0      | 0         | 0      |
| ntfs_event_publisher     | ntfs_event_publisher     | publisher  | 0             | 0      | 0         | 0      |
| ntfs_journal_events      | ntfs_event_publisher     | subscriber | 0             | 0      | 0         | 0      |
| powershell_events        | WindowsEventLogPublisher | subscriber | 0             | 0      | 0         | 0      |
| windows_events           | WindowsEventLogPublisher | subscriber | 0             | 0      | 0         | 0      |
+--------------------------+--------------------------+------------+---------------+--------+-----------+--------+
osquery>
z
Is that top line your command? If so, missing 
--flagfile
t
ahh okay
whats the command to stop, and just return to the shell?
or do i have to kill the pid entirely and open a new shell?
doing this all on a VM, and anytime i need to escalate to an admin shell, i need to swap windows and grab a password
z
Does control-d do it?
t
ahh yes, thanks
whew that took longer than expected....
Copy code
C:\Program Files>osquery\osqueryi.exe --flagfile="C:\Program Files\osquery\osquery.flags"
Using a [1mvirtual database[0m. Need help, type '.help'
osquery>
osquery> select * from osquery_events;
+--------------------------+--------------------------+------------+---------------+--------+-----------+--------+
| name                     | publisher                | type       | subscriptions | events | refreshes | active |
+--------------------------+--------------------------+------------+---------------+--------+-----------+--------+
| WindowsEventLogPublisher | WindowsEventLogPublisher | publisher  | 0             | 0      | 0         | 1      |
| ntfs_event_publisher     | ntfs_event_publisher     | publisher  | 3             | 0      | 0         | 0      |
| ntfs_journal_events      | ntfs_event_publisher     | subscriber | 3             | 0      | 0         | 1      |
| powershell_events        | WindowsEventLogPublisher | subscriber | 0             | 0      | 0         | 0      |
| windows_events           | WindowsEventLogPublisher | subscriber | 0             | 0      | 0         | 0      |
+--------------------------+--------------------------+------------+---------------+--------+-----------+--------+
osquery>
osquery.db was locked by another process
z
Looks like the subscriptions are going now... Does it pick up the changes?
select * from ntfs_journal_events;
t
Copy code
osquery> select * from osquery_events;
+--------------------------+--------------------------+------------+---------------+--------+-----------+--------+
| name                     | publisher                | type       | subscriptions | events | refreshes | active |
+--------------------------+--------------------------+------------+---------------+--------+-----------+--------+
| WindowsEventLogPublisher | WindowsEventLogPublisher | publisher  | 0             | 0      | 0         | 1      |
| ntfs_event_publisher     | ntfs_event_publisher     | publisher  | 3             | 0      | 0         | 0      |
| ntfs_journal_events      | ntfs_event_publisher     | subscriber | 3             | 0      | 0         | 1      |
| powershell_events        | WindowsEventLogPublisher | subscriber | 0             | 0      | 0         | 0      |
| windows_events           | WindowsEventLogPublisher | subscriber | 0             | 0      | 0         | 0      |
+--------------------------+--------------------------+------------+---------------+--------+-----------+--------+
osquery> select * from ntfs_journal_events;
osquery>
nothing
perhaps my yaml config isn't done correctly? file_paths incorrectly defined, or some syntax error in the config. side note: if we want to monitor user directories, could we wildcard where the user name would be? like:
Copy code
C:\Users\*\foo
C:\Users\*\bar
Running this in an enterprise environment where the username is going to be different on every machine.
z
Yes, you ought to be able to wildcard like that for users. I just took another look at your file_paths and I think you probably intend to have wildcards at the end of those paths?
Because I think most of those are directories you've got, so you probably want a trailing 
\%
if you're looking to cover files in those directories.
t
gotcha, okay so would it be like
Copy code
C:\Users\%\foo\%
to wildcard the unique username string, and then everything in some subdirectory within the users home directory?
Copy code
C:\Users\foo\%
or like this? This one seems like it would only work on some user named "foo"
Also, i'm realizing that we are using specific 
file_paths
entries, as well as specific 
exclude_paths
entries. This seems like it would be unnecessary, because if you are doing
Copy code
file_paths
   - A
   - B
   - C
exclude_paths
   - D
   - E
   - F
wouldn't "D" "E" "F" be implicitly excluded since they are not under 
file_paths
?
for reference, here is our current yaml config
Copy code
config:
  options:
    events_expiry: 60
    config_refresh: 600
    host_identifier: instance
    distributed_interval: 60
  decorators:
    load:
      - >-
        SELECT COALESCE((select instance_id FROM ec2_instance_metadata),
        hostname) as hostname FROM system_info;
  file_paths:
    etc:
      - /etc/group
      - /etc/passwd
      - /etc/shadow
      - /etc/services
      - /etc/sudoers
      - /etc/ld.so.preload
      - /etc/ld.so.conf
      - /etc/ld.so.conf.d/%%
      - /etc/pam.d/%%
      - /etc/resolv.conf
      - /etc/modules
      - /etc/hosts
      - /etc/hostname
      - /etc/fstab
      - /etc/rsyslog.conf
    ssh:
      - /root/.ssh/%%
      - /home/%/.ssh/%%
      - /etc/ssh/%%
      - /var/lib/sia/keys/
      - /var/lib/sia/certs/
    logs:
      - /var/log/secure
    docker:
      - /etc/docker/%%
      - /etc/default/docker
      - /etc/docker/daemon.json
      - /usr/bin/containerd
      - /usr/sbin/runc
      - /etc/sysconfig/docker
      - /usr/lib/systemd/system/docker.service
      - /usr/lib/systemd/system/docker.socket
    osquery:
      - /etc/osquery/%%
      - /usr/share/osquery/packs/%%
    firewalls:
      - /etc/sysconfig/iptables
      - /home/y/conf/yakl/%%
      - /etc/yakl/conf/%%
overrides:
  platforms:
    windows:
      options:
        events_expiry: 60
        config_refresh: 600
        host_identifier: instance
        distributed_interval: 60
      decorators:
        load:
          - >-
            SELECT COALESCE((select instance_id FROM ec2_instance_metadata),
            hostname) as hostname FROM system_info;
      file_paths:
        users:
          - 'C:\users\AppData\Roaming'
          - 'C:\users\AppData\Local'
          - 'C:\users\AppData\Local\temp'
          - >-
            C:\users\AppData\Roaming\Microsoft\Windows\Start
            Menu\Programs\Startup
          - 'C:\users\AppData\Roaming\Microsoft\Windows\Start Menu\Programs'
          - 'C:\Users\Default'
        windows:
          - 'C:\Windows'
          - 'C:\Windows\temp'
          - 'C:\Windows\system32\Drivers'
          - 'C:\Windows\SysWOW64\Drivers'
          - 'C:\Windows\system32\GroupPolicy\Machine\Scripts'
          - 'C:\Windows\system32\GroupPolicy\User\Scripts'
          - 'C:\Windows\system32\Wbem'
          - 'C:\Windows\SysWOW64\Wbem'
          - 'C:\Windows\system32\WindowsPowerShell'
          - 'C:\Windows\SysWOW64\WindowsPowerShell'
          - 'C:\Windows\Tasks'
          - 'C:\Windows\system32\Tasks'
          - 'C:\Windows\AppPatch\Custom%'
        ProgramData:
          - 'C:\ProgramData\Microsoft\Windows\Start Menu'
          - 'C:\ProgramData\Microsoft\Windows\Start Menu\Programs'
      exclude_paths:
        windows:
          - 'C:\Windows\system32\DriverStore\Temp\%'
          - 'C:\Windows\system32\wbem\Performance%'
          - 'C:\$WINDOWS.~BT\Sources\%'
          - 'C:\Windows\Installer\%'
          - 'C:\Windows\System32\Tasks\Adobe Acrobat Update Task%'
          - 'C:\Windows\System32\Tasks\Adobe Flash Player Updater%'
          - >-
            C:\Windows\System32\Tasks\OfficeSoftwareProtectionPlatform\SvcRestartTask%
thanks so much for your help with this BTW
10 Views
Open in Slack
PreviousNext
Powered by