https://github.com/osquery/osquery logo
Powered by
Title
t

Ted Dorosheff

01/31/2022, 6:10 PM
hey all, I understand that in mac, the equivalent to `ntfs_journal_events`is 
file_events
, correct? wondering why it is that my 
file_events
table appears active, yet with no subscriptions and no events. I've my FIM category defined via config.
s

sharvil

01/31/2022, 6:14 PM
Do you have 
--enable_file_events=true
in your flag file?
t

Ted Dorosheff

01/31/2022, 6:28 PM
yes
# Server
--tls_server_certs=/var/osquery/certs/cert.pem
--tls_hostname=<http://fleetdm-ui.ouryahoo.com|fleetdm-ui.ouryahoo.com>
--tls_session_reuse=true
--tls_session_timeout=3600
--config_tls_max_attempts=3

# Enrollment
--enroll_secret_path=/var/osquery/secret.txt
--host_identifier=hostname
--enroll_tls_endpoint=/api/v1/osquery/enroll

# Configuration
--config_plugin=filesystem
--config_tls_endpoint=/api/v1/osquery/config
--config_refresh=300
--config_accelerated_refresh=60
--config_check=false
--config_dump=false
--disable_distributed=false
--distributed_plugin=tls
--distributed_interval=10
--distributed_tls_max_attempts=3
--distributed_tls_read_endpoint=/api/v1/osquery/distributed/read
--distributed_tls_write_endpoint=/api/v1/osquery/distributed/write

# Logging
--logger_plugin=tls
--logger_tls_endpoint=/api/v1/osquery/log
--logger_tls_period=10
--logger_tls_period=10
--logger_tls_compress=false
--logger_tls_max_linesize=1048576
--logger_tls_max_lines=2048
--disable_logging=false
--logger_event_type=true
--logger_snapshot_event_type=true
--logger_min_status=0
--logger_min_stderr=0
--logger_stderr=false
--logger_path=/var/osquery/log
--logger_mode=420
--value_max=512
--buffered_log_max=0

# File carving
--disable_carver=false
--carver_start_endpoint=/api/v1/osquery/carve/begin
--carver_continue_endpoint=/api/v1/osquery/carve/block
--carver_block_size=2000000

# Extensions control
--disable_extensions=false

# Daemon control/runtime control
--schedule_splay_percent=10
--schedule_max_drift=60
--schedule_default_interval=3600
--schedule_timeout=0
--pack_refresh_interval=3600
--pack_delimiter=/
--disable_watchdog=false
--watchdog_level=0
--watchdog_memory_limit=0
--watchdog_utilization_limit=0
--watchdog_delay=60
--enable_extensions_watchdog=true
--utc=false
--table_delay=0
--hash_cache_max=500
--hash_delay=20
--disable_caching=false
--disable_hash_cache=false
--read_max=52428800
--force=true
--pidfile=/var/osquery/osqueryd.pidfile

# Backing storage control
--database_path=/var/osquery/osquery.db
--database_dump=false

# Events control
--disable_events=false
--disable_endpointsecurity=false
--events_expiry=3600
--events_optimize=true
--events_max=50000
--enable_fsevents=true
--enable_event_tapping=true
--enable_file_events=true

# Audit control
--disable_audit=false
--audit_persist=true
--audit_allow_config=true
--audit_allow_sockets=true
--audit_allow_process_events=true
s

sharvil

01/31/2022, 6:35 PM
iirc there is no 
--enable_fsevents
flag
also might be worth it to give your terminal full disk access permission on macOS for testing/debugging
t

Ted Dorosheff

01/31/2022, 7:11 PM
yeah i didn't see a fsevents flag either. and ack regarding full disk access
do you have 
file_events
working?
with 
file_paths
defined somewhere?
s

sharvil

01/31/2022, 7:39 PM
Yeah, just tried it, works as expected
Here’s my schedule from the config: 
// Define a schedule of queries:
  "schedule": {
    "file_events": {
      "query": "SELECT * FROM file_events;",
      "removed": false,
      "interval": 2
    }
  },
  "file_paths": {
    "tmp": [
      "/tmp/%%"
    ]
  },
and these options 
"disable_events": "false",
    "enable_file_events": "true"
t

Ted Dorosheff

01/31/2022, 8:36 PM
where do you have the booleans set? in your flags file?
reason i ask is because my flags file looks like this: 
# Server
--tls_server_certs=/var/osquery/certs/cert.pem
--tls_hostname=<http://fleetdm-ui.ouryahoo.com|fleetdm-ui.ouryahoo.com>
--tls_session_reuse=true
--tls_session_timeout=3600
--config_tls_max_attempts=3

# Enrollment
--enroll_secret_path=/var/osquery/secret.txt
--host_identifier=hostname
--enroll_tls_endpoint=/api/v1/osquery/enroll

# Configuration
--config_plugin=filesystem
--config_tls_endpoint=/api/v1/osquery/config
--config_refresh=300
--config_accelerated_refresh=60
--config_check=false
--config_dump=false
--disable_distributed=false
--distributed_plugin=tls
--distributed_interval=10
--distributed_tls_max_attempts=3
--distributed_tls_read_endpoint=/api/v1/osquery/distributed/read
--distributed_tls_write_endpoint=/api/v1/osquery/distributed/write

# Logging
--logger_plugin=tls
--logger_tls_endpoint=/api/v1/osquery/log
--logger_tls_period=10
--logger_tls_period=10
--logger_tls_compress=false
--logger_tls_max_linesize=1048576
--logger_tls_max_lines=2048
--disable_logging=false
--logger_event_type=true
--logger_snapshot_event_type=true
--logger_min_status=0
--logger_min_stderr=0
--logger_stderr=false
--logger_path=/var/osquery/log
--logger_mode=420
--value_max=512
--buffered_log_max=0

# File carving
--disable_carver=false
--carver_start_endpoint=/api/v1/osquery/carve/begin
--carver_continue_endpoint=/api/v1/osquery/carve/block
--carver_block_size=2000000

# Extensions control
--disable_extensions=false

# Daemon control/runtime control
--schedule_splay_percent=10
--schedule_max_drift=60
--schedule_default_interval=3600
--schedule_timeout=0
--pack_refresh_interval=3600
--pack_delimiter=/
--disable_watchdog=false
--watchdog_level=0
--watchdog_memory_limit=0
--watchdog_utilization_limit=0
--watchdog_delay=60
--enable_extensions_watchdog=true
--utc=false
--table_delay=0
--hash_cache_max=500
--hash_delay=20
--disable_caching=false
--disable_hash_cache=false
--read_max=52428800
--force=true
--pidfile=/var/osquery/osqueryd.pidfile

# Backing storage control
--database_path=/var/osquery/osquery.db
--database_dump=false

# Events control
--disable_events=false
--disable_endpointsecurity=false
--events_expiry=3600
--events_optimize=true
--events_max=50000
--enable_fsevents=true
--enable_event_tapping=true
--enable_file_events=true

# Audit control
--disable_audit=false
--audit_persist=true
--audit_allow_config=true
--audit_allow_sockets=true
--audit_allow_process_events=true
i wonder if my format 
--<flag>=<boolean>
is somehow not working, since it looks like yours is in the format 
"<flag>":"<boolean>"
s

Stefano Bonicatti

01/31/2022, 9:09 PM
Those are set inside the config file. I was wondering, maybe the remote config is overriding your?
I would try to test this first with less flags and no remote config
t

Ted Dorosheff

01/31/2022, 9:13 PM
okay thanks Stefano
oh, but i thought 
--config_plugin=filesystem
would disable remote config?
s

Stefano Bonicatti

01/31/2022, 9:19 PM
you’re right, I lost myself in the list ahah. I would still try with with less flags, there might be some adverse interaction I’m not seeing
t

Ted Dorosheff

01/31/2022, 9:20 PM
thats what i'm thinking too. If i started with a blank osquery.flags file, are there any flags that i should include? Do you have a "barebones" osquery flags file?
also, when you said that 
"disable_events": "false",
    "enable_file_events": "true"
would go in the config file, where in config would they go? options?
s

Stefano Bonicatti

01/31/2022, 9:23 PM
I think it should be just a matter of starting osqueryi with 
--enable_file_events=true --disable_events=false --verbose
, and then provide the file paths
t

Ted Dorosheff

01/31/2022, 9:24 PM
okay
s

Stefano Bonicatti

01/31/2022, 9:24 PM
I’ve used verbose to double check that nothing fishy is going on
and yes you could also put them in the config under the “options” key. When you do 
osqueryi --help
, all the flags under “osquery configurations options” can go in the “options” key of the config file. The idea is that those options can be enable or disabled at runtime. Admittedly though that list isn’t always reflecting reality, there a couple of quirks around event publishers if I recall right, where you can’t always disable them to then re-enable them at runtime.
So it’s fine for a test but I would suggest against it
https://github.com/osquery/osquery/issues/6533 this might also be of interest.. and yeah it fell a bit behind.
t

Ted Dorosheff

01/31/2022, 9:37 PM
interesting. I wonder if this is somehow related to another issue i'm having, where i cannot seem to enable logging. 
disable_logging
default boolean is false, yet without any configs whatsoever, it is set to true.
that output was from 
osquery> select * from osquery_flags;
s

Stefano Bonicatti

01/31/2022, 9:41 PM
What about 
config_path
in that table? Have you checked that you don’t have a config file there which sets different options?
t

Ted Dorosheff

01/31/2022, 9:43 PM
my osquery.conf is basically just a big file_paths config. 
{
  "options": {
    "events_expiry": "60",
    "config_refresh": "600",
    "host_identifier": "instance",
    "distributed_interval": 60
  },
  "decorators": {
    "load": [
      "SELECT COALESCE((select instance_id FROM ec2_instance_metadata), hostname) as hostname FROM system_info;"
    ]
  },
  "overrides": {
    "platforms": {
      "windows": {
        "options": {
          "events_expiry": "60",
          "config_refresh": "600",
          "host_identifier": "instance",
          "distributed_interval": "60"
        },
        "decorators": {
          "load": [
            "SELECT COALESCE((select instance_id FROM ec2_instance_metadata), hostname) as hostname FROM system_info;"
          ]
        },
        "file_paths": {
          "users": [
            "C:\\users\\AppData\\Roaming\\%",
            "C:\\users\\AppData\\Local\\%",
            "C:\\users\\AppData\\Local\\temp\\%",
            "C:\\users\\AppData\\Roaming\\Microsoft\\Windows\\StartMenu\\Programs\\Startup\\%",
            "C:\\users\\AppData\\Roaming\\Microsoft\\Windows\\Start Menu\\Programs\\%",
            "C:\\Users\\Default\\%"
          ],
          "Windows": [
            "C:\\Windows\\temp\\%",
            "C:\\Windows\\system32\\Drivers\\%",
            "C:\\Windows\\SysWOW64\\Drivers\\%",
            "C:\\Windows\\system32\\GroupPolicy\\Machine\\Scripts\\%",
            "C:\\Windows\\system32\\GroupPolicy\\User\\Scripts\\%",
            "C:\\Windows\\system32\\Wbem\\%",
            "C:\\Windows\\SysWOW64\\Wbem\\%",
            "C:\\Windows\\system32\\WindowsPowerShell\\%",
            "C:\\Windows\\SysWOW64\\WindowsPowerShell\\%",
            "C:\\Windows\\Tasks\\%",
            "C:\\Windows\\system32\\Tasks\\%",
            "C:\\Windows\\AppPatch\\Custom\\%"
          ],
          "ProgramData": [
            "C:\\ProgramData\\Microsoft\\Windows\\Start Menu\\%",
            "C:\\ProgramData\\Microsoft\\Windows\\Start Menu\\Programs\\%"
          ]
        },
        "exclude_paths": {
          "windows": [
            "C:\\Windows\\system32\\DriverStore\\Temp\\%",
            "C:\\Windows\\system32\\wbem\\Performance\\%",
            "C:\\$WINDOWS.~BT\\Sources\\%",
            "C:\\Windows\\Installer\\%",
            "C:\\Windows\\System32\\Tasks\\Adobe Acrobat Update Task%",
            "C:\\Windows\\System32\\Tasks\\Adobe Flash Player Updater%",
            "C:\\Windows\\System32\\Tasks\\OfficeSoftwareProtectionPlatform\\SvcRestartTask\\%"
          ]
        }
      },
      "linux": {
        "options": {
          "events_expiry": "60",
          "config_refresh": "600",
          "host_identifier": "instance",
          "distributed_interval": "60"
        },
        "decorators": {
          "load": [
            "SELECT COALESCE((select instance_id FROM ec2_instance_metadata), hostname) as hostname FROM system_info;"
          ]
        },
        "file_paths": {
          "etc": [
            "/etc/group",
            "/etc/passwd",
            "/etc/shadow",
            "/etc/services",
            "/etc/sudoers",
            "/etc/ld.so.preload",
            "/etc/ld.so.conf",
            "/etc/ld.so.conf.d/%%",
            "/etc/pam.d/%%",
            "/etc/resolv.conf",
            "/etc/modules",
            "/etc/hosts",
            "/etc/hostname",
            "/etc/fstab",
            "/etc/rsyslog.conf"
          ],
          "ssh": [
            "/root/.ssh/%%",
            "/home/%/.ssh/%%",
            "/etc/ssh/%%",
            "/var/lib/sia/keys/",
            "/var/lib/sia/certs/"
          ],
          "logs": [
            "/var/log/secure"
          ],
          "docker": [
            "/etc/docker/%%",
            "/etc/default/docker",
            "/etc/docker/daemon.json",
            "/usr/bin/containerd",
            "/usr/sbin/runc",
            "/etc/sysconfig/docker",
            "/usr/lib/systemd/system/docker.service",
            "/usr/lib/systemd/system/docker.socket"
          ],
          "osquery": [
            "/etc/osquery/%%",
            "/usr/share/osquery/packs/%%"
          ],
          "firewalls": [
            "/etc/sysconfig/iptables",
            "/home/y/conf/yakl/%%",
            "/etc/yakl/conf/%%"
          ]
        }
      },
      "darwin": {
        "options": {
          "events_expiry": "60",
          "config_refresh": "600",
          "host_identifier": "instance",
          "distributed_interval": "60"
        },
        "decorators": {
          "load": [
            "SELECT COALESCE((select instance_id FROM ec2_instance_metadata), hostname) as hostname FROM system_info;"
          ]
        },
        "file_paths": {
          "System": [
            "/System/Applications/",
            "/System/Library/"
          ],
          "Users": [
            "/Users/%/"
          ],
          "private": [
            "/private/etc/sudoers",
            "/private/etc/ssh/ssh_known_hosts",
            "/private/etc/bashrc"
          ]
        }
      }
    }
  }
}
s

Stefano Bonicatti

01/31/2022, 9:46 PM
oh right wait, if you’re testing with 
osqueryi
logging is force disabled, together with the watchdog.
I keep forgetting that this was changed: https://github.com/osquery/osquery/pull/6621
t

Ted Dorosheff

01/31/2022, 9:47 PM
ahh okay good to know
7 Views
#macosJoin Slack
Powered by