Gregory Storme
12/12/2022, 11:41 AMid: 241686
osquery_host_id: FD723F42-2034-8947-200D-9D1902CF7058
created_at: 2022-12-12 12:12:43
updated_at: 2022-12-12 12:12:43
detail_updated_at: 1970-01-02 01:00:00
node_key: 86UbEE48jOqr3epoiDHg4nLhEwgq390m
uptime: 0
memory: 0
cpu_physical_cores: 0
cpu_logical_cores: 0
primary_ip_id: NULL
distributed_interval: 0
logger_tls_period: 0
config_tls_refresh: 0
label_updated_at: 1970-01-02 01:00:00
last_enrolled_at: 1970-01-02 01:00:00
refetch_requested: 1
team_id: NULL
policy_updated_at: 1970-01-02 01:00:00
orbit_node_key: 86UbEE48jOqr3epoiDHg4nLhEwgq390m
Michal Nicpon
12/12/2022, 12:03 PMGregory Storme
12/12/2022, 12:04 PMMichal Nicpon
12/12/2022, 12:05 PMGregory Storme
12/12/2022, 12:07 PMMichal Nicpon
12/12/2022, 12:09 PMcreated_at
field is the same for all these affected hosts. Does it correspond with the time you upgraded fleet?Gregory Storme
12/12/2022, 12:11 PMMichal Nicpon
12/12/2022, 12:13 PMGregory Storme
12/12/2022, 12:15 PMMichal Nicpon
12/12/2022, 12:23 PM/api/fleet/orbit/enroll
?Gregory Storme
12/12/2022, 12:34 PMMichal Nicpon
12/12/2022, 12:39 PMselect * from hosts where osquery_host_id = "FD723F42-2034-8947-200D-9D1902CF7058"
Gregory Storme
12/12/2022, 12:40 PMid: 243411
osquery_host_id: FD723F42-2034-8947-200D-9D1902CF7058
created_at: 2022-12-12 13:12:56
updated_at: 2022-12-12 13:12:56
detail_updated_at: 1970-01-02 01:00:00
node_key: +hDXpX6Z+97AtEdRIL9X1o7W3iiUOfvv
hostname:
uuid:
platform:
osquery_version:
os_version:
build:
platform_like:
code_name:
uptime: 0
memory: 0
cpu_type:
cpu_subtype:
cpu_brand:
cpu_physical_cores: 0
cpu_logical_cores: 0
hardware_vendor:
hardware_model:
hardware_version:
hardware_serial:
computer_name:
primary_ip_id: NULL
distributed_interval: 0
logger_tls_period: 0
config_tls_refresh: 0
primary_ip:
primary_mac:
label_updated_at: 1970-01-02 01:00:00
last_enrolled_at: 1970-01-02 01:00:00
refetch_requested: 1
team_id: NULL
policy_updated_at: 1970-01-02 01:00:00
public_ip:
orbit_node_key: +hDXpX6Z+97AtEdRIL9X1o7W3iiUOfvv
Michal Nicpon
12/12/2022, 12:42 PMGregory Storme
12/12/2022, 12:45 PMMichal Nicpon
12/12/2022, 12:48 PMGregory Storme
12/12/2022, 1:10 PMMichal Nicpon
12/12/2022, 1:11 PMGregory Storme
12/12/2022, 1:20 PMMichal Nicpon
12/12/2022, 2:47 PMok. I don’t have a way to check the logs on those 2551 hosts, we don’t have centralized logging for the osquery/orbit logsDo you mean you don’t any access at all to these hosts? The logs for even one of the affected hosts would really help here. See docs for where to find orbit logs on various platforms.
Gregory Storme
12/13/2022, 9:09 AMMichal Nicpon
12/13/2022, 9:34 AMosqueryi
on a host and execute the following sql query select uuid from osquery_info
. See if it matches the osquery_host_id
of an affected host in the fleet db
2. Run the following command (osx/linux) ps -eo pid,lstart,command | grep osquery
and see if osquery is either not running, or has a start time that is very recent ie less than 1 hour ago.Gregory Storme
12/13/2022, 10:25 AMosqueryd -S --json "select uuid from osquery_info"
on all of our windows hosts, and 612 uuid's from that query match with an osquery_host_id from those offline hosts in the fleet dbMichal Nicpon
12/13/2022, 12:48 PMGregory Storme
12/13/2022, 12:54 PMMichal Nicpon
12/13/2022, 12:58 PMGet-Process osqueryd | select name,starttime
/api/fleet/orbit/config
Gregory Storme
12/13/2022, 1:05 PMMichal Nicpon
12/13/2022, 1:28 PMC:\Program Files\Orbit\secret-orbit-node-key.txt
exists and is not empty?Gregory Storme
12/13/2022, 1:35 PMMichal Nicpon
12/13/2022, 1:44 PMselect id from hosts where orbit_node_key = "secret"
Replacing secret with the value from the above fileGregory Storme
12/13/2022, 1:47 PMMichal Nicpon
12/13/2022, 1:51 PMGregory Storme
12/13/2022, 1:54 PMMichal Nicpon
12/13/2022, 1:57 PMselect * from hosts
for the problem host and the real host and paste the output?Gregory Storme
12/13/2022, 1:59 PM*************************** 1. row ***************************
id: 2088
osquery_host_id: 89f17842-22be-4b0e-98ac-67efc907a9ba
created_at: 2022-03-21 09:55:53
updated_at: 2022-12-13 14:56:24
detail_updated_at: 2022-12-13 14:56:24
node_key: 4gEbEsUgRIX5yTYrQ7hLhplDbdpOT87U
hostname: MASKED-web01
uuid: 59043F42-0ECC-2981-B532-AD4EEA9D2814
platform: windows
osquery_version: 5.6.0
os_version: Windows Server 2012 R2 Standard
build: 9600
platform_like: windows
code_name: Microsoft Windows Server 2012 R2 Standard
uptime: 2896149000000000
memory: 4294967296
cpu_type: x86_64
cpu_subtype: -1
cpu_brand: Intel(R) Xeon(R) Gold 6142 CPU @ 2.60GHz
cpu_physical_cores: 2
cpu_logical_cores: 2
hardware_vendor: VMware, Inc.
hardware_model: VMware Virtual Platform
hardware_version: -1
hardware_serial: VMware-42 3f 04 59 cc 0e 81 29-b5 32 ad 4e ea 9d 28 14
computer_name: MASKED-WEB01
primary_ip_id: NULL
distributed_interval: 10
logger_tls_period: 10
config_tls_refresh: 60
primary_ip: MASKED
primary_mac: 00:50:56:bf:b6:20
label_updated_at: 2022-12-13 14:55:34
last_enrolled_at: 2022-03-21 09:55:53
refetch_requested: 0
team_id: NULL
policy_updated_at: 2022-12-13 05:40:55
public_ip: MASKED
orbit_node_key: NULL
*************************** 1. row ***************************
id: 290414
osquery_host_id: 59043F42-0ECC-2981-B532-AD4EEA9D2814
created_at: 2022-12-13 14:08:47
updated_at: 2022-12-13 14:08:47
detail_updated_at: 1970-01-02 01:00:00
node_key: 0onOi8pacBhK/lxzUd1NKLRd4JBP02Kf
hostname:
uuid:
platform:
osquery_version:
os_version:
build:
platform_like:
code_name:
uptime: 0
memory: 0
cpu_type:
cpu_subtype:
cpu_brand:
cpu_physical_cores: 0
cpu_logical_cores: 0
hardware_vendor:
hardware_model:
hardware_version:
hardware_serial:
computer_name:
primary_ip_id: NULL
distributed_interval: 0
logger_tls_period: 0
config_tls_refresh: 0
primary_ip:
primary_mac:
label_updated_at: 1970-01-02 01:00:00
last_enrolled_at: 1970-01-02 01:00:00
refetch_requested: 1
team_id: NULL
policy_updated_at: 1970-01-02 01:00:00
public_ip:
orbit_node_key: 0onOi8pacBhK/lxzUd1NKLRd4JBP02Kf
Michal Nicpon
12/13/2022, 2:04 PMAnd the orbit_node_key for that real host is NULL in the fleetdbjust to clarify, is the above output from the same host? How do you know it’s the same host?
Gregory Storme
12/13/2022, 2:08 PMMichal Nicpon
12/13/2022, 2:15 PMGregory Storme
12/13/2022, 7:46 PMRaghavendra Hiremath
12/14/2022, 6:01 AMMichal Nicpon
12/14/2022, 10:32 AMselect * from osquery_flags where name = 'host_identifier'
Gregory Storme
12/14/2022, 10:50 AMMichal Nicpon
12/14/2022, 10:59 AMosquery_host_id
and the uuid
for the real host don’t match. Trying to figure out why that is.command_line_flags
?Gregory Storme
12/14/2022, 11:11 AMcommand_line_flags: {} # requires Fleet's osquery installer
Michal Nicpon
12/14/2022, 11:19 AMGregory Storme
12/14/2022, 11:20 AMconfig:
options:
logger_plugin: tls
disable_carver: true
disable_tables: 'chrome_extensions,firefox_addons'
logger_tls_period: 10
distributed_plugin: tls
disable_distributed: false
logger_tls_endpoint: /api/osquery/log
distributed_interval: 10
carver_disable_function: true
distributed_tls_max_attempts: 3
command_line_flags: {} # requires Fleet's osquery installer
Michal Nicpon
12/14/2022, 11:47 AMGet-CimInstance Win32_process -Filter "name ='osqueryd.exe'" | select CommandLine
Gregory Storme
12/14/2022, 11:53 AMCommandLine : "C:\Program Files\Orbit\bin\osqueryd\windows\stable\osqueryd.exe" "--pidfile=C:\Program Files\Orbit\osque
ry.pid" "--database_path=C:\Program Files\Orbit\osquery.db" --extensions_socket=\\.\pipe\orbit-osquery-ex
tension "--logger_path=C:\Program Files\Orbit\osquery_log" --enroll_secret_env ENROLL_SECRET --host_ident
ifier=uuid --tls_hostname=<http://fleet.x-ops.net|fleet.x-ops.net> --enroll_tls_endpoint=/api/v1/osquery/enroll --config_plu
gin=tls --config_tls_endpoint=/api/v1/osquery/config --config_refresh=60 --disable_distributed=false --di
stributed_plugin=tls --distributed_tls_max_attempts=10 --distributed_tls_read_endpoint=/api/v1/osquery/di
stributed/read --distributed_tls_write_endpoint=/api/v1/osquery/distributed/write --logger_plugin=tls,fil
esystem --logger_tls_endpoint=/api/v1/osquery/log --disable_carver=false --carver_disable_function=false
--carver_start_endpoint=/api/v1/osquery/carve/begin --carver_continue_endpoint=/api/v1/osquery/carve/bloc
k --carver_block_size=2000000 --tls_server_certs "C:\Program Files\Orbit\certs.pem" --force --flagfile "C
:\Program Files\Orbit\osquery.flags"
CommandLine : "C:\Program Files\Orbit\bin\osqueryd\windows\stable\osqueryd.exe" "--pidfile=C:\Program Files\Orbit\osque
ry.pid" "--database_path=C:\Program Files\Orbit\osquery.db" --extensions_socket=\\.\pipe\orbit-osquery-ex
tension "--logger_path=C:\Program Files\Orbit\osquery_log" --enroll_secret_env ENROLL_SECRET --host_ident
ifier=uuid --tls_hostname=<http://fleet.x-ops.net|fleet.x-ops.net> --enroll_tls_endpoint=/api/v1/osquery/enroll --config_plu
gin=tls --config_tls_endpoint=/api/v1/osquery/config --config_refresh=60 --disable_distributed=false --di
stributed_plugin=tls --distributed_tls_max_attempts=10 --distributed_tls_read_endpoint=/api/v1/osquery/di
stributed/read --distributed_tls_write_endpoint=/api/v1/osquery/distributed/write --logger_plugin=tls,fil
esystem --logger_tls_endpoint=/api/v1/osquery/log --disable_carver=false --carver_disable_function=false
--carver_start_endpoint=/api/v1/osquery/carve/begin --carver_continue_endpoint=/api/v1/osquery/carve/bloc
k --carver_block_size=2000000 --tls_server_certs "C:\Program Files\Orbit\certs.pem" --force --flagfile "C
:\Program Files\Orbit\osquery.flags"
Michal Nicpon
12/14/2022, 11:57 AMC:\Program Files\Orbit\osquery.flags
Gregory Storme
12/14/2022, 11:58 AMMichal Nicpon
12/14/2022, 12:09 PM2088
using the Fleet UI. This should trigger osquery running on the host to do a reenrollment.
I still haven’t figured out why there is a discrepancy with the osquery host id, but this should hopefully resolve the issue.
We can confirm it’s resolved by waiting ~1 hr and running the following query on the fleet db
select * from hosts where osquery_host_id = '59043F42-0ECC-2981-B532-AD4EEA9D2814' or uuid = '59043F42-0ECC-2981-B532-AD4EEA9D2814'
We should only get 1 row.Gregory Storme
12/15/2022, 1:16 PMMichal Nicpon
12/15/2022, 1:21 PMGregory Storme
12/15/2022, 1:22 PMMichal Nicpon
12/15/2022, 1:31 PMGregory Storme
12/15/2022, 1:35 PMMichal Nicpon
12/15/2022, 1:40 PMGregory Storme
12/15/2022, 1:41 PMMichal Nicpon
12/15/2022, 3:58 PMGregory Storme
12/19/2022, 5:18 PM