Title
#fleet
j

JL

08/11/2022, 7:53 PM
Hi all. I’m stucking in a problem. My server has wildcard certificate *.test.company.com and the FQDN is final.test.company.com when a try to make a enroll the osquery return
Failed enrollment request to <https://final.test.company.com/api/osquery/enroll> (No node key returned from TLS enroll plugin) retrying...
The same behavior happens without flag --tls_server_cert whats make me think it’s no certificate problem. The certificate was issued in ACM ALB. When a try to make request via curl and send body everything work’s fine and fleet return the node. My osquery.flag is configured with
--force=true
--host_identifier=instance
--verbose=true
--debug
--tls_dump=true

--tls_server_certs=/etc/osquery/fleet.crt

--enroll_secret_env=ENROLL_SECRET
--enroll_tls_endpoint=/api/osquery/enroll

--config_plugin=tls
--config_tls_endpoint=/api/osquery/config
--config_refresh=10

--disable_distributed=false
--distributed_plugin=tls
--distributed_interval=10
--distributed_tls_max_attempts=3
--distributed_tls_read_endpoint=/api/osquery/distributed/read
--distributed_tls_write_endpoint=/api/osquery/distributed/write

--logger_plugin=tls
--logger_tls_endpoint=/api/osquery/log
--logger_tls_period=10

--disable_carver=false
--carver_start_endpoint=/api/osquery/carve/begin
--carver_continue_endpoint=/api/osquery/carve/block
--carver_block_size=2000000
roberto

roberto

08/11/2022, 8:50 PM
hey there! from what I can tell, that error message is not necessarily related to your TLS settings, your host is is having trouble enrolling to the Fleet server. some troubleshooting ideas: 1. check that the env variable`ENROLL_SECRET` is set (as you're providing
--enroll_secret_env=ENROLL_SECRET
) 2. double check that
ENROLL_SECRET
has the correct value 3. Try looking at the
osqueryd
output when you add
--tls_dump
j

JL

08/11/2022, 11:09 PM
Hi Roberto. I checked and everything seems fine. I tried generate package from fleetctl and install deb:
dpkg -i fleet-osquery_1.0.0_amd64.deb
orbit --enroll-secret=XXXXXXXX --fleet-url=<https://final.test.company.com>
/opt/orbit/bin/osqueryd/linux/stable/osqueryd --pidfile=/opt/orbit/osquery.pid --database_path=/opt/orbit/osquery.db --extensions_socket=/opt/orbit/orbit-osquery.em --logger_path=/opt/orbit/osquery_log --enroll_secret_env ENROLL_SECRET --host_identifier=uuid --tls_hostname=final.test.company.com --enroll_tls_endpoint=/api/v1/osquery/enroll --config_plugin=tls --config_tls_endpoint=/api/v1/osquery/config --config_refresh=60 --disable_distributed=false --distributed_plugin=tls --distributed_tls_max_attempts=10 --distributed_tls_read_endpoint=/api/v1/osquery/distributed/read --distributed_tls_write_endpoint=/api/v1/osquery/distributed/write --logger_plugin=tls,filesystem --logger_tls_endpoint=/api/v1/osquery/log --disable_carver=false --carver_disable_function=false --carver_start_endpoint=/api/v1/osquery/carve/begin --carver_continue_endpoint=/api/v1/osquery/carve/block --carver_block_size=2000000 --tls_server_certs /opt/orbit/certs.pem --force --flagfile /opt/orbit/osquery.flags W0811 22:58:52.311698 83 tls_enroll.cpp:101] Failed enrollment request to https://final.test.company.com/api/v1/osquery/enroll (No node key returned from TLS enroll plugin) retrying... W0811 22:58:54.000798 83 tls_enroll.cpp:101] Failed enrollment request to https://final.test.company.com/api/v1/osquery/enroll (No node key returned from TLS enroll plugin) retrying... W0811 22:58:58.710438 83 tls_enroll.cpp:101] Failed enrollment request to https://final.test.company.com/api/v1/osquery/enroll (No node key returned from TLS enroll plugin) retrying... W0811 22:59:08.428079 83 tls_enroll.cpp:101] Failed enrollment request to https://final.test.company.com/api/v1/osquery/enroll (No node key returned from TLS enroll plugin) retrying... W0811 22:59:25.125315 83 tls_enroll.cpp:101] Failed enrollment request to https://final.test.company.com/api/v1/osquery/enroll (No node key returned from TLS enroll plugin) retrying... ^CW0811 22:59:25.588577 83 tls_enroll.cpp:115] Enrollment attempts interrupted due to a shutdown request I0811 22:59:25.589396 201 interface.cpp:137] Registering extension (com.fleetdm.orbit.osquery_extension.v1, 65082, version=, sdk=) Get the same error. one thing that’s different from orbit and my original osquery,flags file is tls_endpoint. in my osquery.flag file I’m using /api/osquery/enroll and orbit use different path /api/v1/osquery/enroll even so the error is the same in both cases.
11:31 AM
I run orbit with —insecure and works. I think this isn't a good idea for production environment.
11:32 AM
I generate new certificate with full FQDN in ACM. My fleet server is behind ALB.
roberto

roberto

08/12/2022, 12:32 PM
thanks for all the details! I still think this is not directly related to your TLS config, the fact that the error happens in the osquery TLS plugin is just a red herring. to give you more context, this error (
No node key returned
) only happens after a successful connection to the Fleet server, if and only if the server returns an empty node key (you can verify my claim by looking at osquery's source code) and this error generally happens with an invalid/empty enroll secret
I run orbit with —insecure and works. I think this isn't a good idea for production environment.
indeed,
--insecure
is not recommended for production usage, as it uses invalid certificates and skips TLS verification can you try running orbit with the exact same parameters you did except for
--insecure
again? One theory I have: it might take a minute or two for the server to start returning the node key for the host, maybe we were exiting the process too early?
j

JL

08/12/2022, 12:58 PM
can you try running orbit with the exact same parameters you did _except for_ --insecure again?
Yes. I receive the same erro
No node key returned from TLS enroll plugin) retrying...
I’m have ALB in front off my fleet. But I generated o self signed certificate for the backend that is behind the ALB. Do you think this could be a problem? I’m thinking run fleet server without SSL. I can’t think nothing more to debug my problem.
2:07 PM
I don’t know if a told this before, but when I get the request body from osquery log (--tls_dump) and make a request using CURL this works fine and return a node key. Ex.:
curl -X POST <https://final.test.company.com/api/v1/osquery/enroll> -d ' {"enroll_secret":"EYhVg","host_identifier":"a97a4442-4352-4e11-8cc2-ab253a712e31","platform_type":"9","host_details":{"os_version":{"_id":"20.04","arch":"x86_64","codename":"focal","major":"20","minor":"04","name":"Ubuntu","patch":"0","pid_with_namespace":"0","platform":"ubuntu","platform_like":"debian","version":"20.04.4 LTS (Focal Fossa)"},"osquery_info":{"build_distro":"centos7","build_platform":"linux","config_hash":"","config_valid":"0","extensions":"active","instance_id":"a97a4442-4352-4e11-8cc2-ab253a712e31","pid":"9","platform_mask":"9","start_time":"1660264491","uuid":"db38421f-0000-0000-a10b-185683d8e894","version":"5.3.0","watcher":"1"},"system_info":{"computer_name":"9e81b139a599","cpu_brand":"Intel(R) Core(TM) i5-8257U CPU @ 1.40GHz\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000","cpu_logical_cores":"4","cpu_physical_cores":"4","cpu_subtype":"142","cpu_type":"x86_64","hardware_model":"","hostname":"9e81b139a599","local_hostname":"9e81b139a599","physical_memory":"12562321408","uuid":"db38421f-0000-0000-a10b-185683d8e894"}}}’
And i receive this return:
{
  "node_key": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
}
Lucas Rodriguez

Lucas Rodriguez

08/12/2022, 2:53 PM
Hi folks. Did you run that
curl
command from the host where Orbit fails to enroll?
zwass

zwass

08/12/2022, 4:34 PM
@JL what response do you see from Fleet in those
tls_dump
logs after it sends the enroll request?
j

JL

08/12/2022, 6:48 PM
Hi folks. Did you run that curl command from the host where Orbit fails to enroll?
@Lucas Rodriguez Yes.
6:51 PM
@zwass
ubuntu20-osquery-tuti_1  | I0812 18:45:14.844377     1 init.cpp:342] osquery initialized [version=4.8.0]
ubuntu20-osquery-tuti_1  | I0812 18:45:14.844758     1 system.cpp:374] Writing osqueryd pid (1) to /var/run/osqueryd.pidfile
ubuntu20-osquery-tuti_1  | I0812 18:45:14.845443     1 extensions.cpp:453] Could not autoload extensions: Cannot open file for reading: /etc/osquery/extensions.load
ubuntu20-osquery-tuti_1  | I0812 18:45:14.845963     1 dispatcher.cpp:78] Adding new service: WatcherRunner (0x5599a55933e8) to thread: 140256932865792 (0x5599a55f6620) in process 1
ubuntu20-osquery-tuti_1  | I0812 18:45:14.847767     7 watcher.cpp:593] osqueryd watcher (1) executing worker (8)
ubuntu20-osquery-tuti_1  | I0812 18:45:14.869949     8 init.cpp:339] osquery worker initialized [watcher=1]
ubuntu20-osquery-tuti_1  | I0812 18:45:14.870151     8 dispatcher.cpp:78] Adding new service: WatcherWatcherRunner (0x563f5cf74038) to thread: 139627301418752 (0x563f5cf6d810) in process 8
ubuntu20-osquery-tuti_1  | I0812 18:45:14.870213     8 rocksdb.cpp:132] Opening RocksDB handle: /var/osquery/osquery.db
ubuntu20-osquery-tuti_1  | I0812 18:45:14.917805     8 dispatcher.cpp:78] Adding new service: ExtensionWatcher (0x563f5d1041b8) to thread: 139626727581440 (0x563f5cf8bd70) in process 8
ubuntu20-osquery-tuti_1  | I0812 18:45:14.917910     8 dispatcher.cpp:78] Adding new service: ExtensionRunnerCore (0x563f5d0360a8) to thread: 139626735974144 (0x563f5cf8bd50) in process 8
ubuntu20-osquery-tuti_1  | I0812 18:45:14.917960     8 tls_enroll.cpp:70] TLSEnrollPlugin requesting a node enroll key from: <https://final.test.company.com/api/v1/osquery/enroll>
ubuntu20-osquery-tuti_1  | I0812 18:45:14.918006    32 interface.cpp:270] Extension manager service starting: /var/osquery/osquery.em
ubuntu20-osquery-tuti_1  | I0812 18:45:14.921299     8 system.cpp:301] Using host identifier: 6b63e2dd-673e-4590-8b58-e7f90f6b404c
ubuntu20-osquery-tuti_1  | I0812 18:45:14.924006     8 smbios_tables.cpp:252] Could not read SMBIOS memory
ubuntu20-osquery-tuti_1  | I0812 18:45:14.925240     8 tls.cpp:255] TLS/HTTPS POST request to URI: <https://final.test.company.com/api/v1/osquery/enroll>
ubuntu20-osquery-tuti_1  | W0812 18:45:16.044317     8 tls_enroll.cpp:77] Failed enrollment request to <https://final.test.company.com/api/v1/osquery/enroll> (No node key returned from TLS enroll plugin) retrying...
ubuntu20-osquery-tuti_1  | I0812 18:45:17.051679     8 smbios_tables.cpp:252] Could not read SMBIOS memory
ubuntu20-osquery-tuti_1  | I0812 18:45:17.053877     8 tls.cpp:255] TLS/HTTPS POST request to URI: <https://final.test.company.com/api/v1/osquery/enroll>
ubuntu20-osquery-tuti_1  | W0812 18:45:17.933704     8 tls_enroll.cpp:77] Failed enrollment request to <https://final.test.company.com/api/v1/osquery/enroll> (No node key returned from TLS enroll plugin) retrying...
ubuntu20-osquery-tuti_1  | I0812 18:45:21.937801     8 smbios_tables.cpp:252] Could not read SMBIOS memory
ubuntu20-osquery-tuti_1  | I0812 18:45:21.939075     8 tls.cpp:255] TLS/HTTPS POST request to URI: <https://final.test.company.com/api/v1/osquery/enroll>
ubuntu20-osquery-tuti_1  | I0812 18:45:22.675884     8 auto_constructed_tables.cpp:97] Removing stale ATC entries
ubuntu20-osquery-tuti_1  | I0812 18:45:22.676192     8 dispatcher.cpp:78] Adding new service: ConfigRefreshRunner (0x563f5cfe1e78) to thread: 139626744366848 (0x563f5d16cd10) in process 8
ubuntu20-osquery-tuti_1  | I0812 18:45:22.676275     8 tls_enroll.cpp:70] TLSEnrollPlugin requesting a node enroll key from: <https://final.test.company.com/api/v1/osquery/enroll>
ubuntu20-osquery-tuti_1  | I0812 18:45:22.678752     8 smbios_tables.cpp:252] Could not read SMBIOS memory
ubuntu20-osquery-tuti_1  | I0812 18:45:22.680732     8 tls.cpp:255] TLS/HTTPS POST request to URI: <https://final.test.company.com/api/v1/osquery/enroll>
ubuntu20-osquery-tuti_1  | {"enroll_secret":"XXXXXXXXXREDACTEDXXXXXXXXXXXXX","host_identifier":"6b63e2dd-673e-4590-8b58-e7f90f6b404c","platform_type":"9","host_details":{"os_version":{"_id":"20.04","arch":"x86_64","codename":"focal","major":"20","minor":"04","name":"Ubuntu","patch":"0","pid_with_namespace":"0","platform":"ubuntu","platform_like":"debian","version":"20.04.2 LTS (Focal Fossa)"},"osquery_info":{"build_distro":"centos7","build_platform":"1","config_hash":"","config_valid":"0","extensions":"active","instance_id":"6b63e2dd-673e-4590-8b58-e7f90f6b404c","pid":"8","platform_mask":"9","start_time":"1660329914","uuid":"e95441f4-0000-0000-a7df-90f533d605dc","version":"4.8.0","watcher":"1"},"system_info":{"computer_name":"8c2d0614551a","cpu_brand":"Intel(R) Core(TM) i5-8257U CPU @ 1.40GHz\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000","cpu_logical_cores":"4","cpu_physical_cores":"4","cpu_subtype":"142","cpu_type":"x86_64","hardware_model":"","hostname":"8c2d0614551a","local_hostname":"8c2d0614551a","physical_memory":"12562321408","uuid":"e95441f4-0000-0000-a7df-90f533d605dc"}}}
ubuntu20-osquery-tuti_1  |
ubuntu20-osquery-tuti_1  | {"enroll_secret":"XXXXXXXXXREDACTEDXXXXXXXXXXXXX","host_identifier":"6b63e2dd-673e-4590-8b58-e7f90f6b404c","platform_type":"9","host_details":{"os_version":{"_id":"20.04","arch":"x86_64","codename":"focal","major":"20","minor":"04","name":"Ubuntu","patch":"0","pid_with_namespace":"0","platform":"ubuntu","platform_like":"debian","version":"20.04.2 LTS (Focal Fossa)"},"osquery_info":{"build_distro":"centos7","build_platform":"1","config_hash":"","config_valid":"0","extensions":"active","instance_id":"6b63e2dd-673e-4590-8b58-e7f90f6b404c","pid":"8","platform_mask":"9","start_time":"1660329914","uuid":"e95441f4-0000-0000-a7df-90f533d605dc","version":"4.8.0","watcher":"1"},"system_info":{"computer_name":"8c2d0614551a","cpu_brand":"Intel(R) Core(TM) i5-8257U CPU @ 1.40GHz\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000","cpu_logical_cores":"4","cpu_physical_cores":"4","cpu_subtype":"142","cpu_type":"x86_64","hardware_model":"","hostname":"8c2d0614551a","local_hostname":"8c2d0614551a","physical_memory":"12562321408","uuid":"e95441f4-0000-0000-a7df-90f533d605dc"}}}
...
ubuntu20-osquery-tuti_1  | {"enroll_secret":"XXXXXXXXXREDACTEDXXXXXXXXXXXXX","host_identifier":"6b63e2dd-673e-4590-8b58-e7f90f6b404c","platform_type":"9","host_details":{"os_version":{"_id":"20.04","arch":"x86_64","codename":"focal","major":"20","minor":"04","name":"Ubuntu","patch":"0","pid_with_namespace":"0","platform":"ubuntu","platform_like":"debian","version":"20.04.2 LTS (Focal Fossa)"},"osquery_info":{"build_distro":"centos7","build_platform":"1","config_hash":"","config_valid":"0","extensions":"active","instance_id":"6b63e2dd-673e-4590-8b58-e7f90f6b404c","pid":"8","platform_mask":"9","start_time":"1660329914","uuid":"e95441f4-0000-0000-a7df-90f533d605dc","version":"4.8.0","watcher":"1"},"system_info":{"computer_name":"8c2d0614551a","cpu_brand":"Intel(R) Core(TM) i5-8257U CPU @ 1.40GHz\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000","cpu_logical_cores":"4","cpu_physicalW0812 18:45:23.544463     8 tls_enroll.cpp:77] Failed enrollment request to <https://final.test.company.com/api/v1/osquery/enroll> (No node key returned from TLS enroll plugin) retrying...
ubuntu20-osquery-tuti_1  | I0812 18:45:24.556339     8 smbios_tables.cpp:252] Could not read SMBIOS memory
ubuntu20-osquery-tuti_1  | I0812 18:45:24.558943     8 tls.cpp:255] TLS/HTTPS POST request to URI: <https://final.test.company.com/api/v1/osquery/enroll>
ubuntu20-osquery-tuti_1  | W0812 18:45:25.489326     8 tls_enroll.cpp:77] Failed enrollment request to <https://final.test.company.com/api/v1/osquery/enroll> (No node key returned from TLS enroll plugin) retrying...
ubuntu20-osquery-tuti_1  | I0812 18:45:29.495386     8 smbios_tables.cpp:252] Could not read SMBIOS memory
ubuntu20-osquery-tuti_1  | I0812 18:45:29.498567     8 tls.cpp:255] TLS/HTTPS POST request to URI: <https://final.test.company.com/api/v1/osquery/enroll>
ubuntu20-osquery-tuti_1  | I0812 18:45:30.327318     8 tls.cpp:255] TLS/HTTPS POST request to URI: <https://final.test.company.com/api/v1/osquery/config>
ubuntu20-osquery-tuti_1  | I0812 18:45:31.055061     8 tls_enroll.cpp:70] TLSEnrollPlugin requesting a node enroll key from: <https://final.test.company.com/api/v1/osquery/enroll>
ubuntu20-osquery-tuti_1  | I0812 18:45:31.061553     8 smbios_tables.cpp:252] Could not read SMBIOS memory
ubuntu20-osquery-tuti_1  | I0812 18:45:31.063885     8 tls.cpp:255] TLS/HTTPS POST request to URI: <https://final.test.company.com/api/v1/osquery/enroll>
ubuntu20-osquery-tuti_1  | W0812 18:45:31.925765     8 tls_enroll.cpp:77] Failed enrollment request to <https://final.test.company.com/api/v1/osquery/enroll> (No node key returned from TLS enroll plugin) retrying...
ubuntu20-osquery-tuti_1  | I0812 18:45:32.679306    39 config.cpp:1206] Refreshing configuration state
ubuntu20-osquery-tuti_1  | I0812 18:45:32.931587     8 smbios_tables.cpp:252] Could not read SMBIOS memory
ubuntu20-osquery-tuti_1  | I0812 18:45:32.934620     8 tls.cpp:255] TLS/HTTPS POST request to URI: <https://final.test.company.com/api/v1/osquery/enroll>
ubuntu20-osquery-tuti_1  | _cores":"4","cpu_subtype":"142","cpu_type":"x86_64","hardware_model":"","hostname":"8c2d0614551a","local_hostname":"8c2d0614551a","physical_memory":"12562321408","uuid":"e95441f4-0000-0000-a7df-90f533d605dc"}}}
ubuntu20-osquery-tuti_1  |
ubuntu20-osquery-tuti_1  | {"enroll_secret":"XXXXXXXXXREDACTEDXXXXXXXXXXXXX","host_identifier":"6b63e2dd-673e-4590-8b58-e7f90f6b404c","platform_type":"9","host_details":{"os_version":{"_id":"20.04","arch":"x86_64","codename":"focal","major":"20","minor":"04","name":"Ubuntu","patch":"0","pid_with_namespace":"0","platform":"ubuntu","platform_like":"debian","version":"20.04.2 LTS (Focal Fossa)"},"osquery_info":{"build_distro":"centos7","build_platform":"1","config_hash":"","config_valid":"0","extensions":"active","instance_id":"6b63e2dd-673e-4590-8b58-e7f90f6b404c","pid":"8","platform_mask":"9","start_time":"1660329914","uuid":"e95441f4-0000-0000-a7df-90f533d605dc","version":"4.8.0","watcher":"1"},"system_info":{"computer_name":"8c2d0614551a","cpu_brand":"Intel(R) Core(TM) i5-8257U CPU @ 1.40GHz\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000","cpu_logical_cores":"4","cpu_physical_cores":"4","cpu_subtype":"142","cpu_type":"x86_64","hardware_model":"","hostname":"8c2d0614551a","local_hostname":"8c2d0614551a","physical_memory":"12562321408","uuid":"e95441f4-0000-0000-a7df-90f533d605dc"}}}
ubuntu20-osquery-tuti_1  |
ubuntu20-osquery-tuti_1  | {"enroll_secret":"XXXXXXXXXREDACTEDXXXXXXXXXXXXX","host_identifier":"6b63e2dd-673e-4590-8b58-e7f90f6b404c","platform_type":"9","host_details":{"os_version":{"_id":"20.04","arch":"x86_64","codename":"focal","major":"20","minor":"04","name":"Ubuntu","patch":"0","pid_with_namespace":"0","platform":"ubuntu","platform_like":"debian","version":"20.04.2 LTS (Focal Fossa)"},"osquery_info":{"build_distro":"centos7","build_platform":"1","config_hash":"","config_valid":"0","extensions":"active","instance_id":"6b63e2dd-673e-4590-8b58-e7f90f6b404c","pid":"8","platform_mask":"9","start_time":"1660329914","uuid":"e95441f4-0000-0000-a7df-90f533d605dc","version":"4.8.0","watcher":"1"},"system_info":{"computer_name":"8c2d0614551a","cpu_brand":"Intel(R) Core(TM) i5-8257U CPU @ 1.40GHz\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000","cpu_logical_cores":"4","cpu_physical_cores":"4","cpu_subtype":"142","cpu_type":"x86_64","hardware_model":"","hostname":"8c2d0614551a","local_hostname":"8c2d0614551a","physical_memory":"12562321408","uuid":"e95441f4-0000-0000-a7df-90f533d605dc"}}}
ubuntu20-osquery-tuti_1  |
ubuntu20-osquery-tuti_1  | {"node_key":""}
ubuntu20-osquery-tuti_1  |
Kathy Satterlee

Kathy Satterlee

08/17/2022, 4:34 PM
Hey @JL! Sorry about the delay in getting back to you here. Are you still having trouble getting that host enrolled?
j

JL

08/27/2022, 8:55 PM
Hi @Kathy Satterlee I discovered the problem. The fleet was behind alb with aws waf enabled and i was using self signed certificate in fleet server. I removed self signed certificate from fleet and the enrollment start working properly returning the node key. After that I was getting second error in the endpoint /api/v1/osquery/distributed/write that's returning 403. I made troubleshooting and checked payload grpc. I can’t find the rule and make correct adjustments yet. But after i remove waf everything works fine.