Title
#fleet
s

sanjaykcse

03/04/2021, 6:13 AM
Hi , I have some packs on fleet and dumping the logs with option (--tls_dump=true and --verbose) , I could see the pack queries being executed on the host . When I send " live query " SELECT * FROM osquery_schedule , the result is "No results found" . Do the packs pushed from fleet do not get listed in osquery_schedule and if that cases , how to check the blacklisted queries ?
zwass

zwass

03/04/2021, 4:26 PM
Packs pushed from Fleet do get listed in osquery_schedule. Are you certain your live query is hitting the same host that is executing the scheduled queries? Can you see the live query request and response in those tls_dump logs?
s

sanjaykcse

03/04/2021, 5:44 PM
Yes. I could see the live query hitting the same host. Below is the config --enroll_secret_path=/etc/osquery/cert/secret --tls_server_certs=/etc/osquery/cert --tls_hostname=<server>:8080 --database_path=/tmp/osquery.db --pidfile=/tmp/osquery.pid --host_identifier=uuid --force=true --config_plugin=tls --config_tls_endpoint=/api/v1/osquery/config --config_tls_refresh=30 --disable_distributed=false --distributed_interval=3 --distributed_plugin=tls --config_tls_max_attempts=3 --distributed_tls_max_attempts=3 --distributed_tls_read_endpoint=/api/v1/osquery/distributed/read --distributed_tls_write_endpoint=/api/v1/osquery/distributed/write --hash_delay=20 --pack_refresh_interval=60 --schedule_splay_percent=20 --table_delay=20 --enroll_tls_endpoint=/api/v1/osquery/enroll --disable_watchdog=false --watchdog_delay=120 --watchdog_level=1 --watchdog_memory_limit=100 --watchdog_utilization_limit=5 --logger_plugin=filesystem --logger_path=/tmp
zwass

zwass

03/04/2021, 5:58 PM
This sounds like possibly a bug in osquery? If you can provide logs that show the same osquery process (1) receiving a config with a nonempty schedule and (2) returning an empty result set for
select * from osquery
then you could file an issue for osquery and it could be investigated.
s

sanjaykcse

03/04/2021, 6:45 PM
--------------------- I0304 18:52:42.263559 13256 tls.cpp:254] TLS/HTTPS POST request to URI: https://ec2-54-75-78-244.eu-west-1.compute.amazonaws.com:8080/api/v1/osquery/distributed/read I0304 18:52:53.134891 13256 tls.cpp:254] TLS/HTTPS POST request to URI: https://ec2-54-75-78-244.eu-west-1.compute.amazonaws.com:8080/api/v1/osquery/distributed/read I0304 18:52:53.744874 13256 distributed.cpp:121] Executing distributed query: kolide_distributed_query_4: SELECT * FROM osquery_schedule I0304 18:52:53.745616 13256 tls.cpp:254] TLS/HTTPS POST request to URI: https://ec2-54-75-78-244.eu-west-1.compute.amazonaws.com:8080/api/v1/osquery/distributed/write I0304 18:53:00.308753 13242 config.cpp:1205] Refreshing configuration state I0304 18:53:00.309159 13242 tls.cpp:254] TLS/HTTPS POST request to URI: https://ec2-54-75-78-244.eu-west-1.compute.amazonaws.com:8080/api/v1/osquery/config I0304 18:53:04.472676 13256 tls.cpp:254] TLS/HTTPS POST request to URI: https://ec2-54-75-78-244.eu-west-1.compute.amazonaws.com:8080/api/v1/osquery/distributed/read I0304 18:53:15.224409 13256 tls.cpp:254] TLS/HTTPS POST request to URI: https://ec2-54-75-78-244.eu-west-1.compute.amazonaws.com:8080/api/v1/osquery/distributed/read I0304 18:53:25.871349 13256 tls.cpp:254] TLS/HTTPS POST request to URI: https://ec2-54-75-78-244.eu-west-1.compute.amazonaws.com:8080/api/v1/osquery/distributed/read I0304 18:53:30.915606 13242 config.cpp:1205] Refreshing configuration state I0304 18:53:30.916021 13242 tls.cpp:254] TLS/HTTPS POST request to URI: https://ec2-54-75-78-244.eu-west-1.compute.amazonaws.com:8080/api/v1/osquery/config I0304 18:53:36.478948 13256 tls.cpp:254] TLS/HTTPS POST request to URI: https://ec2-54-75-78-244.eu-west-1.compute.amazonaws.com:8080/api/v1/osquery/distributed/read I0304 18:53:47.100065 13256 tls.cpp:254] TLS/HTTPS POST request to URI: https://ec2-54-75-78-244.eu-west-1.compute.amazonaws.com:8080/api/v1/osquery/distributed/read I0304 18:53:57.731017 13256 tls.cpp:254] TLS/HTTPS POST request to URI: https://ec2-54-75-78-244.eu-west-1.compute.amazonaws.com:8080/api/v1/osquery/distributed/read "distributed_tls_max_attempts": 3,   "pack_delimiter": "/"  },  "packs": {   "normal": {    "queries": {     "osquery": {      "query": "SELECT * FROM osquery_info",      "interval": 10,      "platform": "",      "version": "",      "removed": true,      "shard": 1     },     "processes": {      "query": "SELECT * FROM processes",      "interval": 10,      "platform": "",      "version": "",      "removed": true,      "shard": 2     }    }   }  } } {} {"node_key":"B4YMnz3q4v5/Q7zSb1Na9i0IUBInAW0F"} {  "queries": {} } {"node_key":"B4YMnz3q4v5/Q7zSb1Na9i0IUBInAW0F"} {  "queries": {} } {"node_key":"B4YMnz3q4v5/Q7zSb1Na9i0IUBInAW0F"} {  "decorators": {   "load": [    "SELECT uuid AS host_uuid FROM system_info;",    "SELECT hostname AS hostname FROM system_info;"   ]  },  "options": {   "disable_distributed": false,   "distributed_interval": 10,   "distributed_plugin": "tls",   "distributed_tls_max_attempts": 3,   "pack_delimiter": "/"  },  "packs": {   "normal": {    "queries": {     "osquery": {      "query": "SELECT * FROM osquery_info",      "interval": 10,      "platform": "",      "version": "",      "removed": true,      "shard": 1     },     "processes": {      "query": "SELECT * FROM processes",      "interval": 10,      "platform": "",      "version": "",      "removed": true,      "shard": 2     }    }   }  } } {"node_key":"B4YMnz3q4v5/Q7zSb1Na9i0IUBInAW0F"} {  "queries": {} } {"node_key":"B4YMnz3q4v5/Q7zSb1Na9i0IUBInAW0F"} {  "queries": {} } {"node_key":"B4YMnz3q4v5/Q7zSb1Na9i0IUBInAW0F"} {  "queries": {   "kolide_distributed_query_4": "SELECT * FROM osquery_schedule"  } } {"queries":{"kolide_distributed_query_4":[]},"statuses":{"kolide_distributed_query_4":0},"messages":{"kolide_distributed_query_4":""},"node_key":"B4YMnz3q4v5/Q7zSb1Na9i0IUBInAW0F"} {} {"node_key":"B4YMnz3q4v5/Q7zSb1Na9i0IUBInAW0F"} {  "decorators": {   "load": [    "SELECT uuid AS host_uuid FROM system_info;",    "SELECT hostname AS hostname FROM system_info;"   ]  },  "options": {   "disable_distributed": false,   "distributed_interval": 10,   "distributed_plugin": "tls",   "distributed_tls_max_attempts": 3,   "pack_delimiter": "/"  },  "packs": {   "normal": {    "queries": {     "osquery": {      "query": "SELECT * FROM osquery_info",      "interval": 10,      "platform": "",      "version": "",      "removed": true,      "shard": 1     },     "processes": {      "query": "SELECT * FROM processes",      "interval": 10,      "platform": "",      "version": "",      "removed": true,      "shard": 2     }    }   }  } } {"node_key":"B4YMnz3q4v5/Q7zSb1Na9i0IUBInAW0F"} {  "queries": {} } {"node_key":"B4YMnz3q4v5/Q7zSb1Na9i0IUBInAW0F"} {  "queries": {} } {"node_key":"B4YMnz3q4v5/Q7zSb1Na9i0IUBInAW0F"} {  "queries": {} } {"node_key":"B4YMnz3q4v5/Q7zSb1Na9i0IUBInAW0F"} {  "decorators": {   "load": [    "SELECT uuid AS host_uuid FROM system_info;",    "SELECT hostname AS hostname FROM system_info;"   ]  },  "options": {   "disable_distributed": false,   "distributed_interval": 10,   "distributed_plugin": "tls",   "distributed_tls_max_attempts": 3,   "pack_delimiter": "/"  },  "packs": {   "normal": {    "queries": {     "osquery": {      "query": "SELECT * FROM osquery_info",      "interval": 10,      "platform": "",      "version": "",      "removed": true,      "shard": 1     },     "processes": {      "query": "SELECT * FROM processes",      "interval": 10,      "platform": "",      "version": "",      "removed": true,      "shard": 2     }    }   }  } } {"node_key":"B4YMnz3q4v5/Q7zSb1Na9i0IUBInAW0F"} {  "queries": {} } {"node_key":"B4YMnz3q4v5/Q7zSb1Na9i0IUBInAW0F"} {  "queries": {} } {"node_key":"B4YI0304 18:54:01.665854 13242 config.cpp:1205] Refreshing configuration state I0304 18:54:01.666270 13242 tls.cpp:254] TLS/HTTPS POST request to URI: https://ec2-54-75-78-244.eu-west-1.compute.amazonaws.com:8080/api/v1/osquery/config
6:47 PM
ubuntu@ip-172-31-39-251:~$ ls -lt /tmp/osquery_result  -rw-r--r-- 1 root root 0 Mar 4 18:31 /tmp/osquery_result I am not getting query pack results on fleet either
6:49 PM
I am using osquery version 4.5.1
zwass

zwass

03/04/2021, 7:50 PM
Ah, you are setting
shard
to 1 and 2, meaning only 1 or 2% of the hosts that receive the query will actually schedule it.
7:50 PM
So it's most likely that the host just isn't scheduling the query.
s

sanjaykcse

03/05/2021, 3:49 AM
Yes , I had set Shard to 1 . changing thisto 0, works . Thanks @zwass .Appreciate your help
6:19 AM
There two issue here1. setting shard ------resolved 2. If pass config option for not sending the logs over TLS --logger_plugin=filesystem --logger_path=/tmp The demoen stops sending results for scheduled query to fleet .Is there any option to just send only result logs to fleet and not the status logs ?
6:36 AM
If logger is configured for TLS and due to any reason fleet is offline , the osquery daemon keeps on sending the log/results even the TLS connection is not there Error sending status to logger: Request error: Failed to connect to ec2-54-75-78-244.eu-west-1.compute.amazonaws.com:8080: Connection refused After few hours , the osqueryd starts crashing . to prevent the issue I chose to send the logs to local system . But it stops sending results also .
9:34 AM
would setting "--buffered_log_max" help preventing such crashes , if the it due to growing buffer when not able to set log/results over TLS ?
zwass

zwass

03/05/2021, 3:37 PM
Osquery should not crash... If it does that's a bug.
s

sanjaykcse

03/06/2021, 8:12 AM
Seem I has been hitting the bug https://github.com/osquery/osquery/issues/6887 . which is fix 15 days back. I have cherry-picked the changes .