https://github.com/osquery/osquery logo
Join Slack
Powered by
Hi all. I’m stucking in a problem. My server has w...
# fleet
j
Hi all. I’m stucking in a problem. My server has wildcard certificate *.test.company.com and the FQDN is final.test.company.com when a try to make a enroll the osquery return 
Failed enrollment request to <https://final.test.company.com/api/osquery/enroll> (No node key returned from TLS enroll plugin) retrying...
The same behavior happens without flag --tls_server_cert whats make me think it’s no certificate problem. The certificate was issued in ACM ALB. When a try to make request via curl and send body everything work’s fine and fleet return the node. My osquery.flag is configured with
Copy code
--force=true
--host_identifier=instance
--verbose=true
--debug
--tls_dump=true

--tls_server_certs=/etc/osquery/fleet.crt

--enroll_secret_env=ENROLL_SECRET
--enroll_tls_endpoint=/api/osquery/enroll

--config_plugin=tls
--config_tls_endpoint=/api/osquery/config
--config_refresh=10

--disable_distributed=false
--distributed_plugin=tls
--distributed_interval=10
--distributed_tls_max_attempts=3
--distributed_tls_read_endpoint=/api/osquery/distributed/read
--distributed_tls_write_endpoint=/api/osquery/distributed/write

--logger_plugin=tls
--logger_tls_endpoint=/api/osquery/log
--logger_tls_period=10

--disable_carver=false
--carver_start_endpoint=/api/osquery/carve/begin
--carver_continue_endpoint=/api/osquery/carve/block
--carver_block_size=2000000
r
hey there! from what I can tell, that error message is not necessarily related to your TLS settings, your host is is having trouble enrolling to the Fleet server. some troubleshooting ideas: 1. check that the env variable`ENROLL_SECRET` is set (as you're providing 
--enroll_secret_env=ENROLL_SECRET
) 2. double check that 
ENROLL_SECRET
has the correct value 3. Try looking at the 
osqueryd
output when you add 
--tls_dump
j
Hi Roberto. I checked and everything seems fine. I tried generate package from fleetctl and install deb: 
dpkg -i fleet-osquery_1.0.0_amd64.deb
orbit --enroll-secret=XXXXXXXX --fleet-url=<https://final.test.company.com>
/opt/orbit/bin/osqueryd/linux/stable/osqueryd --pidfile=/opt/orbit/osquery.pid --database_path=/opt/orbit/osquery.db --extensions_socket=/opt/orbit/orbit-osquery.em --logger_path=/opt/orbit/osquery_log --enroll_secret_env ENROLL_SECRET --host_identifier=uuid --tls_hostname=final.test.company.com --enroll_tls_endpoint=/api/v1/osquery/enroll --config_plugin=tls --config_tls_endpoint=/api/v1/osquery/config --config_refresh=60 --disable_distributed=false --distributed_plugin=tls --distributed_tls_max_attempts=10 --distributed_tls_read_endpoint=/api/v1/osquery/distributed/read --distributed_tls_write_endpoint=/api/v1/osquery/distributed/write --logger_plugin=tls,filesystem --logger_tls_endpoint=/api/v1/osquery/log --disable_carver=false --carver_disable_function=false --carver_start_endpoint=/api/v1/osquery/carve/begin --carver_continue_endpoint=/api/v1/osquery/carve/block --carver_block_size=2000000 --tls_server_certs /opt/orbit/certs.pem --force --flagfile /opt/orbit/osquery.flags W0811 225852.311698 83 tls_enroll.cpp:101] Failed enrollment request to https://final.test.company.com/api/v1/osquery/enroll (No node key returned from TLS enroll plugin) retrying... W0811 225854.000798 83 tls_enroll.cpp:101] Failed enrollment request to https://final.test.company.com/api/v1/osquery/enroll (No node key returned from TLS enroll plugin) retrying... W0811 225858.710438 83 tls_enroll.cpp:101] Failed enrollment request to https://final.test.company.com/api/v1/osquery/enroll (No node key returned from TLS enroll plugin) retrying... W0811 225908.428079 83 tls_enroll.cpp:101] Failed enrollment request to https://final.test.company.com/api/v1/osquery/enroll (No node key returned from TLS enroll plugin) retrying... W0811 225925.125315 83 tls_enroll.cpp:101] Failed enrollment request to https://final.test.company.com/api/v1/osquery/enroll (No node key returned from TLS enroll plugin) retrying... ^CW0811 225925.588577 83 tls_enroll.cpp:115] Enrollment attempts interrupted due to a shutdown request I0811 225925.589396 201 interface.cpp:137] Registering extension (com.fleetdm.orbit.osquery_extension.v1, 65082, version=, sdk=) Get the same error. one thing that’s different from orbit and my original osquery,flags file is tls_endpoint. in my osquery.flag file I’m using /api/osquery/enroll and orbit use different path /api/v1/osquery/enroll even so the error is the same in both cases.
I run orbit with —insecure and works. I think this isn't a good idea for production environment.
I generate new certificate with full FQDN in ACM. My fleet server is behind ALB.
r
thanks for all the details! I still think this is not directly related to your TLS config, the fact that the error happens in the osquery TLS plugin is just a red herring. to give you more context, this error (
No node key returned
) only happens after a successful connection to the Fleet server, if and only if the server returns an empty node key (you can verify my claim by looking at osquery's source code) and this error generally happens with an invalid/empty enroll secret
I run orbit with —insecure and works. I think this isn't a good idea for production environment.
indeed, 
--insecure
is not recommended for production usage, as it uses invalid certificates and skips TLS verification can you try running orbit with the exact same parameters you did except for 
--insecure
again? One theory I have: it might take a minute or two for the server to start returning the node key for the host, maybe we were exiting the process too early?
j
can you try running orbit with the exact same parameters you did _except for_ --insecure again?
Yes. I receive the same erro 
No node key returned from TLS enroll plugin) retrying...
I’m have ALB in front off my fleet. But I generated o self signed certificate for the backend that is behind the ALB. Do you think this could be a problem? I’m thinking run fleet server without SSL. I can’t think nothing more to debug my problem.
I don’t know if a told this before, but when I get the request body from osquery log (--tls_dump) and make a request using CURL this works fine and return a node key. Ex.:
Copy code
curl -X POST <https://final.test.company.com/api/v1/osquery/enroll> -d ' {"enroll_secret":"EYhVg","host_identifier":"a97a4442-4352-4e11-8cc2-ab253a712e31","platform_type":"9","host_details":{"os_version":{"_id":"20.04","arch":"x86_64","codename":"focal","major":"20","minor":"04","name":"Ubuntu","patch":"0","pid_with_namespace":"0","platform":"ubuntu","platform_like":"debian","version":"20.04.4 LTS (Focal Fossa)"},"osquery_info":{"build_distro":"centos7","build_platform":"linux","config_hash":"","config_valid":"0","extensions":"active","instance_id":"a97a4442-4352-4e11-8cc2-ab253a712e31","pid":"9","platform_mask":"9","start_time":"1660264491","uuid":"db38421f-0000-0000-a10b-185683d8e894","version":"5.3.0","watcher":"1"},"system_info":{"computer_name":"9e81b139a599","cpu_brand":"Intel(R) Core(TM) i5-8257U CPU @ 1.40GHz\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000","cpu_logical_cores":"4","cpu_physical_cores":"4","cpu_subtype":"142","cpu_type":"x86_64","hardware_model":"","hostname":"9e81b139a599","local_hostname":"9e81b139a599","physical_memory":"12562321408","uuid":"db38421f-0000-0000-a10b-185683d8e894"}}}’
And i receive this return:
Copy code
{
  "node_key": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
}
l
Hi folks. Did you run that 
curl
command from the host where Orbit fails to enroll?
z
@JL what response do you see from Fleet in those 
tls_dump
logs after it sends the enroll request?
j
Hi folks. Did you run that curl command from the host where Orbit fails to enroll?
@Lucas Rodriguez Yes.
@zwass
Copy code
ubuntu20-osquery-tuti_1  | I0812 18:45:14.844377     1 init.cpp:342] osquery initialized [version=4.8.0]
ubuntu20-osquery-tuti_1  | I0812 18:45:14.844758     1 system.cpp:374] Writing osqueryd pid (1) to /var/run/osqueryd.pidfile
ubuntu20-osquery-tuti_1  | I0812 18:45:14.845443     1 extensions.cpp:453] Could not autoload extensions: Cannot open file for reading: /etc/osquery/extensions.load
ubuntu20-osquery-tuti_1  | I0812 18:45:14.845963     1 dispatcher.cpp:78] Adding new service: WatcherRunner (0x5599a55933e8) to thread: 140256932865792 (0x5599a55f6620) in process 1
ubuntu20-osquery-tuti_1  | I0812 18:45:14.847767     7 watcher.cpp:593] osqueryd watcher (1) executing worker (8)
ubuntu20-osquery-tuti_1  | I0812 18:45:14.869949     8 init.cpp:339] osquery worker initialized [watcher=1]
ubuntu20-osquery-tuti_1  | I0812 18:45:14.870151     8 dispatcher.cpp:78] Adding new service: WatcherWatcherRunner (0x563f5cf74038) to thread: 139627301418752 (0x563f5cf6d810) in process 8
ubuntu20-osquery-tuti_1  | I0812 18:45:14.870213     8 rocksdb.cpp:132] Opening RocksDB handle: /var/osquery/osquery.db
ubuntu20-osquery-tuti_1  | I0812 18:45:14.917805     8 dispatcher.cpp:78] Adding new service: ExtensionWatcher (0x563f5d1041b8) to thread: 139626727581440 (0x563f5cf8bd70) in process 8
ubuntu20-osquery-tuti_1  | I0812 18:45:14.917910     8 dispatcher.cpp:78] Adding new service: ExtensionRunnerCore (0x563f5d0360a8) to thread: 139626735974144 (0x563f5cf8bd50) in process 8
ubuntu20-osquery-tuti_1  | I0812 18:45:14.917960     8 tls_enroll.cpp:70] TLSEnrollPlugin requesting a node enroll key from: <https://final.test.company.com/api/v1/osquery/enroll>
ubuntu20-osquery-tuti_1  | I0812 18:45:14.918006    32 interface.cpp:270] Extension manager service starting: /var/osquery/osquery.em
ubuntu20-osquery-tuti_1  | I0812 18:45:14.921299     8 system.cpp:301] Using host identifier: 6b63e2dd-673e-4590-8b58-e7f90f6b404c
ubuntu20-osquery-tuti_1  | I0812 18:45:14.924006     8 smbios_tables.cpp:252] Could not read SMBIOS memory
ubuntu20-osquery-tuti_1  | I0812 18:45:14.925240     8 tls.cpp:255] TLS/HTTPS POST request to URI: <https://final.test.company.com/api/v1/osquery/enroll>
ubuntu20-osquery-tuti_1  | W0812 18:45:16.044317     8 tls_enroll.cpp:77] Failed enrollment request to <https://final.test.company.com/api/v1/osquery/enroll> (No node key returned from TLS enroll plugin) retrying...
ubuntu20-osquery-tuti_1  | I0812 18:45:17.051679     8 smbios_tables.cpp:252] Could not read SMBIOS memory
ubuntu20-osquery-tuti_1  | I0812 18:45:17.053877     8 tls.cpp:255] TLS/HTTPS POST request to URI: <https://final.test.company.com/api/v1/osquery/enroll>
ubuntu20-osquery-tuti_1  | W0812 18:45:17.933704     8 tls_enroll.cpp:77] Failed enrollment request to <https://final.test.company.com/api/v1/osquery/enroll> (No node key returned from TLS enroll plugin) retrying...
ubuntu20-osquery-tuti_1  | I0812 18:45:21.937801     8 smbios_tables.cpp:252] Could not read SMBIOS memory
ubuntu20-osquery-tuti_1  | I0812 18:45:21.939075     8 tls.cpp:255] TLS/HTTPS POST request to URI: <https://final.test.company.com/api/v1/osquery/enroll>
ubuntu20-osquery-tuti_1  | I0812 18:45:22.675884     8 auto_constructed_tables.cpp:97] Removing stale ATC entries
ubuntu20-osquery-tuti_1  | I0812 18:45:22.676192     8 dispatcher.cpp:78] Adding new service: ConfigRefreshRunner (0x563f5cfe1e78) to thread: 139626744366848 (0x563f5d16cd10) in process 8
ubuntu20-osquery-tuti_1  | I0812 18:45:22.676275     8 tls_enroll.cpp:70] TLSEnrollPlugin requesting a node enroll key from: <https://final.test.company.com/api/v1/osquery/enroll>
ubuntu20-osquery-tuti_1  | I0812 18:45:22.678752     8 smbios_tables.cpp:252] Could not read SMBIOS memory
ubuntu20-osquery-tuti_1  | I0812 18:45:22.680732     8 tls.cpp:255] TLS/HTTPS POST request to URI: <https://final.test.company.com/api/v1/osquery/enroll>
ubuntu20-osquery-tuti_1  | {"enroll_secret":"XXXXXXXXXREDACTEDXXXXXXXXXXXXX","host_identifier":"6b63e2dd-673e-4590-8b58-e7f90f6b404c","platform_type":"9","host_details":{"os_version":{"_id":"20.04","arch":"x86_64","codename":"focal","major":"20","minor":"04","name":"Ubuntu","patch":"0","pid_with_namespace":"0","platform":"ubuntu","platform_like":"debian","version":"20.04.2 LTS (Focal Fossa)"},"osquery_info":{"build_distro":"centos7","build_platform":"1","config_hash":"","config_valid":"0","extensions":"active","instance_id":"6b63e2dd-673e-4590-8b58-e7f90f6b404c","pid":"8","platform_mask":"9","start_time":"1660329914","uuid":"e95441f4-0000-0000-a7df-90f533d605dc","version":"4.8.0","watcher":"1"},"system_info":{"computer_name":"8c2d0614551a","cpu_brand":"Intel(R) Core(TM) i5-8257U CPU @ 1.40GHz\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000","cpu_logical_cores":"4","cpu_physical_cores":"4","cpu_subtype":"142","cpu_type":"x86_64","hardware_model":"","hostname":"8c2d0614551a","local_hostname":"8c2d0614551a","physical_memory":"12562321408","uuid":"e95441f4-0000-0000-a7df-90f533d605dc"}}}
ubuntu20-osquery-tuti_1  |
ubuntu20-osquery-tuti_1  | {"enroll_secret":"XXXXXXXXXREDACTEDXXXXXXXXXXXXX","host_identifier":"6b63e2dd-673e-4590-8b58-e7f90f6b404c","platform_type":"9","host_details":{"os_version":{"_id":"20.04","arch":"x86_64","codename":"focal","major":"20","minor":"04","name":"Ubuntu","patch":"0","pid_with_namespace":"0","platform":"ubuntu","platform_like":"debian","version":"20.04.2 LTS (Focal Fossa)"},"osquery_info":{"build_distro":"centos7","build_platform":"1","config_hash":"","config_valid":"0","extensions":"active","instance_id":"6b63e2dd-673e-4590-8b58-e7f90f6b404c","pid":"8","platform_mask":"9","start_time":"1660329914","uuid":"e95441f4-0000-0000-a7df-90f533d605dc","version":"4.8.0","watcher":"1"},"system_info":{"computer_name":"8c2d0614551a","cpu_brand":"Intel(R) Core(TM) i5-8257U CPU @ 1.40GHz\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000","cpu_logical_cores":"4","cpu_physical_cores":"4","cpu_subtype":"142","cpu_type":"x86_64","hardware_model":"","hostname":"8c2d0614551a","local_hostname":"8c2d0614551a","physical_memory":"12562321408","uuid":"e95441f4-0000-0000-a7df-90f533d605dc"}}}
...
ubuntu20-osquery-tuti_1  | {"enroll_secret":"XXXXXXXXXREDACTEDXXXXXXXXXXXXX","host_identifier":"6b63e2dd-673e-4590-8b58-e7f90f6b404c","platform_type":"9","host_details":{"os_version":{"_id":"20.04","arch":"x86_64","codename":"focal","major":"20","minor":"04","name":"Ubuntu","patch":"0","pid_with_namespace":"0","platform":"ubuntu","platform_like":"debian","version":"20.04.2 LTS (Focal Fossa)"},"osquery_info":{"build_distro":"centos7","build_platform":"1","config_hash":"","config_valid":"0","extensions":"active","instance_id":"6b63e2dd-673e-4590-8b58-e7f90f6b404c","pid":"8","platform_mask":"9","start_time":"1660329914","uuid":"e95441f4-0000-0000-a7df-90f533d605dc","version":"4.8.0","watcher":"1"},"system_info":{"computer_name":"8c2d0614551a","cpu_brand":"Intel(R) Core(TM) i5-8257U CPU @ 1.40GHz\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000","cpu_logical_cores":"4","cpu_physicalW0812 18:45:23.544463     8 tls_enroll.cpp:77] Failed enrollment request to <https://final.test.company.com/api/v1/osquery/enroll> (No node key returned from TLS enroll plugin) retrying...
ubuntu20-osquery-tuti_1  | I0812 18:45:24.556339     8 smbios_tables.cpp:252] Could not read SMBIOS memory
ubuntu20-osquery-tuti_1  | I0812 18:45:24.558943     8 tls.cpp:255] TLS/HTTPS POST request to URI: <https://final.test.company.com/api/v1/osquery/enroll>
ubuntu20-osquery-tuti_1  | W0812 18:45:25.489326     8 tls_enroll.cpp:77] Failed enrollment request to <https://final.test.company.com/api/v1/osquery/enroll> (No node key returned from TLS enroll plugin) retrying...
ubuntu20-osquery-tuti_1  | I0812 18:45:29.495386     8 smbios_tables.cpp:252] Could not read SMBIOS memory
ubuntu20-osquery-tuti_1  | I0812 18:45:29.498567     8 tls.cpp:255] TLS/HTTPS POST request to URI: <https://final.test.company.com/api/v1/osquery/enroll>
ubuntu20-osquery-tuti_1  | I0812 18:45:30.327318     8 tls.cpp:255] TLS/HTTPS POST request to URI: <https://final.test.company.com/api/v1/osquery/config>
ubuntu20-osquery-tuti_1  | I0812 18:45:31.055061     8 tls_enroll.cpp:70] TLSEnrollPlugin requesting a node enroll key from: <https://final.test.company.com/api/v1/osquery/enroll>
ubuntu20-osquery-tuti_1  | I0812 18:45:31.061553     8 smbios_tables.cpp:252] Could not read SMBIOS memory
ubuntu20-osquery-tuti_1  | I0812 18:45:31.063885     8 tls.cpp:255] TLS/HTTPS POST request to URI: <https://final.test.company.com/api/v1/osquery/enroll>
ubuntu20-osquery-tuti_1  | W0812 18:45:31.925765     8 tls_enroll.cpp:77] Failed enrollment request to <https://final.test.company.com/api/v1/osquery/enroll> (No node key returned from TLS enroll plugin) retrying...
ubuntu20-osquery-tuti_1  | I0812 18:45:32.679306    39 config.cpp:1206] Refreshing configuration state
ubuntu20-osquery-tuti_1  | I0812 18:45:32.931587     8 smbios_tables.cpp:252] Could not read SMBIOS memory
ubuntu20-osquery-tuti_1  | I0812 18:45:32.934620     8 tls.cpp:255] TLS/HTTPS POST request to URI: <https://final.test.company.com/api/v1/osquery/enroll>
ubuntu20-osquery-tuti_1  | _cores":"4","cpu_subtype":"142","cpu_type":"x86_64","hardware_model":"","hostname":"8c2d0614551a","local_hostname":"8c2d0614551a","physical_memory":"12562321408","uuid":"e95441f4-0000-0000-a7df-90f533d605dc"}}}
ubuntu20-osquery-tuti_1  |
ubuntu20-osquery-tuti_1  | {"enroll_secret":"XXXXXXXXXREDACTEDXXXXXXXXXXXXX","host_identifier":"6b63e2dd-673e-4590-8b58-e7f90f6b404c","platform_type":"9","host_details":{"os_version":{"_id":"20.04","arch":"x86_64","codename":"focal","major":"20","minor":"04","name":"Ubuntu","patch":"0","pid_with_namespace":"0","platform":"ubuntu","platform_like":"debian","version":"20.04.2 LTS (Focal Fossa)"},"osquery_info":{"build_distro":"centos7","build_platform":"1","config_hash":"","config_valid":"0","extensions":"active","instance_id":"6b63e2dd-673e-4590-8b58-e7f90f6b404c","pid":"8","platform_mask":"9","start_time":"1660329914","uuid":"e95441f4-0000-0000-a7df-90f533d605dc","version":"4.8.0","watcher":"1"},"system_info":{"computer_name":"8c2d0614551a","cpu_brand":"Intel(R) Core(TM) i5-8257U CPU @ 1.40GHz\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000","cpu_logical_cores":"4","cpu_physical_cores":"4","cpu_subtype":"142","cpu_type":"x86_64","hardware_model":"","hostname":"8c2d0614551a","local_hostname":"8c2d0614551a","physical_memory":"12562321408","uuid":"e95441f4-0000-0000-a7df-90f533d605dc"}}}
ubuntu20-osquery-tuti_1  |
ubuntu20-osquery-tuti_1  | {"enroll_secret":"XXXXXXXXXREDACTEDXXXXXXXXXXXXX","host_identifier":"6b63e2dd-673e-4590-8b58-e7f90f6b404c","platform_type":"9","host_details":{"os_version":{"_id":"20.04","arch":"x86_64","codename":"focal","major":"20","minor":"04","name":"Ubuntu","patch":"0","pid_with_namespace":"0","platform":"ubuntu","platform_like":"debian","version":"20.04.2 LTS (Focal Fossa)"},"osquery_info":{"build_distro":"centos7","build_platform":"1","config_hash":"","config_valid":"0","extensions":"active","instance_id":"6b63e2dd-673e-4590-8b58-e7f90f6b404c","pid":"8","platform_mask":"9","start_time":"1660329914","uuid":"e95441f4-0000-0000-a7df-90f533d605dc","version":"4.8.0","watcher":"1"},"system_info":{"computer_name":"8c2d0614551a","cpu_brand":"Intel(R) Core(TM) i5-8257U CPU @ 1.40GHz\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000","cpu_logical_cores":"4","cpu_physical_cores":"4","cpu_subtype":"142","cpu_type":"x86_64","hardware_model":"","hostname":"8c2d0614551a","local_hostname":"8c2d0614551a","physical_memory":"12562321408","uuid":"e95441f4-0000-0000-a7df-90f533d605dc"}}}
ubuntu20-osquery-tuti_1  |
ubuntu20-osquery-tuti_1  | {"node_key":""}
ubuntu20-osquery-tuti_1  |
k
Hey @JL! Sorry about the delay in getting back to you here. Are you still having trouble getting that host enrolled?
j
Hi @Kathy Satterlee I discovered the problem. The fleet was behind alb with aws waf enabled and i was using self signed certificate in fleet server. I removed self signed certificate from fleet and the enrollment start working properly returning the node key. After that I was getting second error in the endpoint /api/v1/osquery/distributed/write that's returning 403. I made troubleshooting and checked payload grpc. I can’t find the rule and make correct adjustments yet. But after i remove waf everything works fine.
128 Views
Open in Slack
PreviousNext
Powered by