https://github.com/osquery/osquery logo
Powered by
#kolide
Title
d

Dan Achin

11/04/2020, 5:59 PM
hi, I'm getting the following warning in osqueryd.INFO, osqueryd.WARNING and osqueryd.ERROR logs on my Fleet servers. I tried searching the internet for it, but didn't find anything. Also, it's happening the same exact # of time in each log, which seems strange to me - why would those 3 logs all log this warning? Anyone know about this? We aren't running any queries on the kernel_extensions table. In fact, we are still very early in our rollout so we have just a handful of basic queries scheduled so we have data to look at in Splunk. I'm talking very basic, like "SELECT * FROM osquery_info" kind of basic. osqueryd.WARNING:E1102 165455.280033 10849 scheduler.cpp:101] Error executing scheduled query macos_kextstat: no such table: kernel_extensions
z

zwass

11/04/2020, 6:03 PM
That's a macos only table. Are you scheduling it on other systems?
d

Dan Achin

11/04/2020, 6:16 PM
are you saying osquery_info is macos only?
we aren't running anything with macos_kextstat
sorry, i mean kernel_extensions
let me go verify
z

zwass

11/04/2020, 6:18 PM
I'm just looking at your error message
d

Dan Achin

11/04/2020, 6:18 PM
right
we are literally running like 5 queries across 2 packs
they are things like "SELECT * FROM users;"
is it possible a query is scheduled that we can't see in the UI?
z

zwass

11/04/2020, 7:57 PM
It would be a bug if there is a scheduled query not available in the UI. Are you possibly using a filesystem configuration?
d

Dan Achin

11/04/2020, 8:54 PM
Not that I can tell (I didn't set any of this up). I see that we have two packs scheduled, each with a couple of queries. Here are all of the queries we run: SELECT * FROM osquery_info SELECT * FROM users SELECT * FROM osquery_schedule WHERE denylisted='1'; SELECT * FROM osquery_schedule SELECT path, label, program_arguments, inetd_compatibility, root_directory FROM launchd  - Mac only SELECT * FROM crontab  - Mac / linux SELECT * FROM scheduled_tasks - Windows
now...one think I see is that the two packs are applied to all hosts, but then inside the pack we have the queries defined for specific OS when warranted. Could that be the issue? Maybe we can't apply them to 'All Hosts'
@zwass, do you think that's the issue ^ ?
z

zwass

11/05/2020, 6:05 AM
None of those seem to be querying that table. Can you look at osquery output with 
--verbose --tls_dump
and see if your host is receiving the config you expect?
d

Dan Achin

11/05/2020, 5:29 PM
Thanks @zwass. let me try that after some meetings. thanks
👍 1
Copy code
C02W28G1HTD6:infrastructure dachin$ sudo osqueryd --ephemeral --disable_database --disable_logging --verbose --tls_dump
I1105 10:26:38.275341 139969984 init.cpp:343] osquery initialized [version=4.4.0]
I1105 10:26:38.276684 139969984 extensions.cpp:383] Could not autoload extensions: Failed reading: /var/osquery/extensions.load
I1105 10:26:38.276727 139969984 dispatcher.cpp:77] Adding new service: WatcherRunner (0x7f863d705288) to thread: 0x70000098e000 (0x7f863d7045a0) in process 59573
I1105 10:26:38.277495 10018816 watcher.cpp:585] osqueryd watcher (59573) executing worker (59574)
I1105 10:26:38.295192 286705088 init.cpp:340] osquery worker initialized [watcher=59573]
I1105 10:26:38.298854 286705088 dispatcher.cpp:77] Adding new service: WatcherWatcherRunner (0x7ff8fac002b8) to thread: 0x7000083df000 (0x7ff8fac000c0) in process 59574
I1105 10:26:38.298998 286705088 dispatcher.cpp:77] Adding new service: ExtensionWatcher (0x7ff8fac00498) to thread: 0x700008462000 (0x7ff8fac003c0) in process 59574
I1105 10:26:38.299039 286705088 dispatcher.cpp:77] Adding new service: ExtensionRunnerCore (0x7ff8fac006a8) to thread: 0x7000084e5000 (0x7ff8fac00140) in process 59574
I1105 10:26:38.299090 286705088 auto_constructed_tables.cpp:96] Removing stale ATC entries
W1105 10:26:38.299268 286705088 init.cpp:602] Error reading config: config file does not exist: /var/osquery/osquery.conf
I1105 10:26:38.299289 286705088 events.cpp:866] Event publisher not enabled: openbsm: Publisher disabled via configuration
I1105 10:26:38.299297 286705088 events.cpp:866] Event publisher not enabled: scnetwork: Publisher not used
I1105 10:26:38.299302 286705088 events.cpp:866] Event publisher not enabled: event_tapping: Publisher disabled via configuration
I1105 10:26:38.299435 286705088 events.cpp:1125] Error registering subscriber: socket_events: Subscriber disabled via configuration
I1105 10:26:38.300441 286705088 main.cpp:103] Not starting the distributed query service: Distributed query service not enabled.
I1105 10:26:38.300451 139886592 events.cpp:785] Starting event publisher run loop: diskarbitration
I1105 10:26:38.300454 140423168 events.cpp:785] Starting event publisher run loop: fsevents
I1105 10:26:38.300475 286705088 dispatcher.cpp:77] Adding new service: SchedulerRunner (0x7ff8fad18018) to thread: 0x7000086f1000 (0x7ff8fad131a0) in process 59574
I1105 10:26:38.300468 140959744 events.cpp:785] Starting event publisher run loop: iokit
I1105 10:26:38.499281 139350016 interface.cpp:268] Extension manager service starting: /var/osquery/osquery.em
Is that what you were looking for?
z

zwass

11/05/2020, 6:39 PM
I see 
Error reading config: config file does not exist: /var/osquery/osquery.conf
. Is this the same configuration you were using when you saw the error message (besides 
--verbose --tls_dump
)?
Not seeing any of the requests/responses with the Fleet server. Those are possibly below.
d

Dan Achin

11/05/2020, 6:46 PM
ya, we aren't using that config file as far as i know, i always thought it just pulled what it needed from fleet based on the flags file
that's where the output stops...
z

zwass

11/05/2020, 6:49 PM
Is there a flagfile you usually provide when connecting to the Fleet server?
d

Dan Achin

11/05/2020, 6:49 PM
I've always thought it strange that I never see anything in debug that looks like the client talking to fleet. But this client is registered in fleet...I just ran a query against it actually
yep
sec
Copy code
C02W28G1HTD6:infrastructure dachin$ cat /etc/osquery/osquery.flags 
--enroll_secret_path=/etc/osquery/fleet_secret
--tls_hostname=<http://fleet-corp.nonprod.ppops.net:443|fleet-corp.nonprod.ppops.net:443>
--host_identifier=uuid
--enroll_tls_endpoint=/api/v1/osquery/enroll
--config_plugin=tls
--config_tls_endpoint=/api/v1/osquery/config
--config_refresh=3600
--disable_distributed=false
--distributed_plugin=tls
--distributed_interval=60
--distributed_tls_max_attempts=3
--distributed_tls_read_endpoint=/api/v1/osquery/distributed/read
--distributed_tls_write_endpoint=/api/v1/osquery/distributed/write
--logger_plugin=tls
--logger_tls_endpoint=/api/v1/osquery/log
--logger_tls_period=10
z

zwass

11/05/2020, 6:51 PM
When debugging like this you need to provide the same flags as the service is using. Perhaps that is 
osqueryd --flagfile /etc/osquery/osquery.flags --verbose --tls_dump
?
d

Dan Achin

11/05/2020, 6:51 PM
oh!
right
thanks
much better
what am I looking for specifically? 🙂
something with the kernel_extentions table perhaps
Copy code
"packs": {
    "AllHostsTest": {
      "queries": {
        "AllUsers": {
          "query": "SELECT * FROM users",
          "interval": 86400,
          "platform": "",
          "version": "",
          "snapshot": true,
          "removed": false
        },
        "Check Denied Queries": {
          "query": "SELECT * FROM osquery_schedule WHERE denylisted='1';\n",
          "interval": 600,
          "platform": "",
          "version": "",
          "snapshot": true,
          "removed": false
        },
        "Check osquery_schedule": {
          "query": "SELECT * FROM osquery_schedule",
          "interval": 3600,
          "platform": "",
          "version": "3.3.1",
          "snapshot": true,
          "removed": false
        },
        "T1053.004 - osx_launchd": {
          "query": "SELECT path, label, program_arguments, inetd_compatibility, root_directory\nFROM launchd",
          "interval": 3600,
          "platform": "darwin",
          "version": "3.3.1",
          "snapshot": true,
          "removed": false
        },
        "osquery_info": {
          "query": "SELECT * FROM osquery_info",
          "interval": 86400,
          "platform": "",
          "version": "3.3.1",
          "snapshot": true,
          "removed": false
        }
      }
    },
    "T1053 - Scheduled Task or Job": {
      "queries": {
        "T1053.003 - Mac/Linux Cron Tasks": {
          "query": "SELECT * FROM crontab",
          "interval": 86400,
          "platform": "darwin,linux",
          "removed": true
        },
        "T1053.004 - osx_launchd": {
          "query": "SELECT path, label, program_arguments, inetd_compatibility, root_directory\nFROM launchd",
          "interval": 86400,
          "platform": "darwin",
          "removed": true
        },
        "T1053.005 - T1053.002 - Windows Scheduled Tasks": {
          "query": "SELECT * FROM scheduled_tasks",
          "interval": 86400,
          "platform": "windows",
          "removed": true
        }
      }
    }
  }
}
I'm assuming 'removed: true' means it was filtered out?
z

zwass

11/05/2020, 7:00 PM
So this host is logging the execution error for kernel_extensions? Because I'm not seeing a kernel_extensions query in here.
d

Dan Achin

11/05/2020, 7:13 PM
right...i don't know which hosts are generating that error, only that it's all through our fleet logs. those packs that are applied to this host are the same packs applied to all 287 nodes we have registered...they are the only packs we have defined
maybe i'm thinking about this wrong. I'm looking at the logs on the fleet server itself under /var/log/osquery
osquery.ERROR, osquery.INFO and osquery.WARNING. Are these entries from osquery on fleet itself trying to run packs. I don't see that we have our fleet servers themselves registered in Fleet
z

zwass

11/05/2020, 7:33 PM
Those log files are actually on the Fleet server? Are you running osquery on the Fleet server perhaps from some previous testing? Fleet itself would not output to those paths.
d

Dan Achin

11/05/2020, 7:37 PM
yes....they are on the fleet server
mystery solved...pretty sure osquery is in our fleet puppet profile
strange though...it's not registered anywhere so why is it trying to run that stuff
z

zwass

11/05/2020, 7:38 PM
Puppet is pushing a config with queries?
d

Dan Achin

11/05/2020, 7:39 PM
nah, we don't actually push any packs out via puppet
ok, I think i have what I need. we'll want to register the fleet instances themselves with fleet at some point - they are just as important as other servers. must just be this partial config that's causing the errors
Doh! I found it. the fleet servers have a couple packs defined in /etc/osquery/osquery.conf. One of them is select * from kernel_extensions
👍 1
2 Views
Powered by