JL
08/11/2022, 7:53 PMFailed enrollment request to <https://final.test.company.com/api/osquery/enroll> (No node key returned from TLS enroll plugin) retrying...
The same behavior happens without flag --tls_server_cert whats make me think it’s no certificate problem. The certificate was issued in ACM ALB.
When a try to make request via curl and send body everything work’s fine and fleet return the node.
My osquery.flag is configured with
--force=true
--host_identifier=instance
--verbose=true
--debug
--tls_dump=true
--tls_server_certs=/etc/osquery/fleet.crt
--enroll_secret_env=ENROLL_SECRET
--enroll_tls_endpoint=/api/osquery/enroll
--config_plugin=tls
--config_tls_endpoint=/api/osquery/config
--config_refresh=10
--disable_distributed=false
--distributed_plugin=tls
--distributed_interval=10
--distributed_tls_max_attempts=3
--distributed_tls_read_endpoint=/api/osquery/distributed/read
--distributed_tls_write_endpoint=/api/osquery/distributed/write
--logger_plugin=tls
--logger_tls_endpoint=/api/osquery/log
--logger_tls_period=10
--disable_carver=false
--carver_start_endpoint=/api/osquery/carve/begin
--carver_continue_endpoint=/api/osquery/carve/block
--carver_block_size=2000000
roberto
08/11/2022, 8:50 PM--enroll_secret_env=ENROLL_SECRET
)
2. double check that ENROLL_SECRET
has the correct value
3. Try looking at the osqueryd
output when you add --tls_dump
JL
08/11/2022, 11:09 PMdpkg -i fleet-osquery_1.0.0_amd64.deb
orbit --enroll-secret=XXXXXXXX --fleet-url=<https://final.test.company.com>
/opt/orbit/bin/osqueryd/linux/stable/osqueryd --pidfile=/opt/orbit/osquery.pid --database_path=/opt/orbit/osquery.db --extensions_socket=/opt/orbit/orbit-osquery.em --logger_path=/opt/orbit/osquery_log --enroll_secret_env ENROLL_SECRET --host_identifier=uuid --tls_hostname=final.test.company.com --enroll_tls_endpoint=/api/v1/osquery/enroll --config_plugin=tls --config_tls_endpoint=/api/v1/osquery/config --config_refresh=60 --disable_distributed=false --distributed_plugin=tls --distributed_tls_max_attempts=10 --distributed_tls_read_endpoint=/api/v1/osquery/distributed/read --distributed_tls_write_endpoint=/api/v1/osquery/distributed/write --logger_plugin=tls,filesystem --logger_tls_endpoint=/api/v1/osquery/log --disable_carver=false --carver_disable_function=false --carver_start_endpoint=/api/v1/osquery/carve/begin --carver_continue_endpoint=/api/v1/osquery/carve/block --carver_block_size=2000000 --tls_server_certs /opt/orbit/certs.pem --force --flagfile /opt/orbit/osquery.flags
W0811 225852.311698 83 tls_enroll.cpp:101] Failed enrollment request to https://final.test.company.com/api/v1/osquery/enroll (No node key returned from TLS enroll plugin) retrying...
W0811 225854.000798 83 tls_enroll.cpp:101] Failed enrollment request to https://final.test.company.com/api/v1/osquery/enroll (No node key returned from TLS enroll plugin) retrying...
W0811 225858.710438 83 tls_enroll.cpp:101] Failed enrollment request to https://final.test.company.com/api/v1/osquery/enroll (No node key returned from TLS enroll plugin) retrying...
W0811 225908.428079 83 tls_enroll.cpp:101] Failed enrollment request to https://final.test.company.com/api/v1/osquery/enroll (No node key returned from TLS enroll plugin) retrying...
W0811 225925.125315 83 tls_enroll.cpp:101] Failed enrollment request to https://final.test.company.com/api/v1/osquery/enroll (No node key returned from TLS enroll plugin) retrying...
^CW0811 225925.588577 83 tls_enroll.cpp:115] Enrollment attempts interrupted due to a shutdown request
I0811 225925.589396 201 interface.cpp:137] Registering extension (com.fleetdm.orbit.osquery_extension.v1, 65082, version=, sdk=)
Get the same error.
one thing that’s different from orbit and my original osquery,flags file is tls_endpoint. in my osquery.flag file I’m using /api/osquery/enroll and orbit use different path /api/v1/osquery/enroll even so the error is the same in both cases.roberto
08/12/2022, 12:32 PMNo node key returned
) only happens after a successful connection to the Fleet server, if and only if the server returns an empty node key (you can verify my claim by looking at osquery's source code) and this error generally happens with an invalid/empty enroll secret
I run orbit with —insecure and works. I think this isn't a good idea for production environment.indeed,
--insecure
is not recommended for production usage, as it uses invalid certificates and skips TLS verification
can you try running orbit with the exact same parameters you did except for --insecure
again? One theory I have: it might take a minute or two for the server to start returning the node key for the host, maybe we were exiting the process too early?JL
08/12/2022, 12:58 PMcan you try running orbit with the exact same parameters you did _except for_ --insecure again?
Yes. I receive the same erro No node key returned from TLS enroll plugin) retrying...
I’m have ALB in front off my fleet. But I generated o self signed certificate for the backend that is behind the ALB. Do you think this could be a problem? I’m thinking run fleet server without SSL. I can’t think nothing more to debug my problem.curl -X POST <https://final.test.company.com/api/v1/osquery/enroll> -d ' {"enroll_secret":"EYhVg","host_identifier":"a97a4442-4352-4e11-8cc2-ab253a712e31","platform_type":"9","host_details":{"os_version":{"_id":"20.04","arch":"x86_64","codename":"focal","major":"20","minor":"04","name":"Ubuntu","patch":"0","pid_with_namespace":"0","platform":"ubuntu","platform_like":"debian","version":"20.04.4 LTS (Focal Fossa)"},"osquery_info":{"build_distro":"centos7","build_platform":"linux","config_hash":"","config_valid":"0","extensions":"active","instance_id":"a97a4442-4352-4e11-8cc2-ab253a712e31","pid":"9","platform_mask":"9","start_time":"1660264491","uuid":"db38421f-0000-0000-a10b-185683d8e894","version":"5.3.0","watcher":"1"},"system_info":{"computer_name":"9e81b139a599","cpu_brand":"Intel(R) Core(TM) i5-8257U CPU @ 1.40GHz\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000","cpu_logical_cores":"4","cpu_physical_cores":"4","cpu_subtype":"142","cpu_type":"x86_64","hardware_model":"","hostname":"9e81b139a599","local_hostname":"9e81b139a599","physical_memory":"12562321408","uuid":"db38421f-0000-0000-a10b-185683d8e894"}}}’
And i receive this return:
{
"node_key": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
}
Lucas Rodriguez
08/12/2022, 2:53 PMcurl
command from the host where Orbit fails to enroll?zwass
tls_dump
logs after it sends the enroll request?JL
08/12/2022, 6:48 PMHi folks. Did you run that curl command from the host where Orbit fails to enroll?
@Lucas Rodriguez Yes.ubuntu20-osquery-tuti_1 | I0812 18:45:14.844377 1 init.cpp:342] osquery initialized [version=4.8.0]
ubuntu20-osquery-tuti_1 | I0812 18:45:14.844758 1 system.cpp:374] Writing osqueryd pid (1) to /var/run/osqueryd.pidfile
ubuntu20-osquery-tuti_1 | I0812 18:45:14.845443 1 extensions.cpp:453] Could not autoload extensions: Cannot open file for reading: /etc/osquery/extensions.load
ubuntu20-osquery-tuti_1 | I0812 18:45:14.845963 1 dispatcher.cpp:78] Adding new service: WatcherRunner (0x5599a55933e8) to thread: 140256932865792 (0x5599a55f6620) in process 1
ubuntu20-osquery-tuti_1 | I0812 18:45:14.847767 7 watcher.cpp:593] osqueryd watcher (1) executing worker (8)
ubuntu20-osquery-tuti_1 | I0812 18:45:14.869949 8 init.cpp:339] osquery worker initialized [watcher=1]
ubuntu20-osquery-tuti_1 | I0812 18:45:14.870151 8 dispatcher.cpp:78] Adding new service: WatcherWatcherRunner (0x563f5cf74038) to thread: 139627301418752 (0x563f5cf6d810) in process 8
ubuntu20-osquery-tuti_1 | I0812 18:45:14.870213 8 rocksdb.cpp:132] Opening RocksDB handle: /var/osquery/osquery.db
ubuntu20-osquery-tuti_1 | I0812 18:45:14.917805 8 dispatcher.cpp:78] Adding new service: ExtensionWatcher (0x563f5d1041b8) to thread: 139626727581440 (0x563f5cf8bd70) in process 8
ubuntu20-osquery-tuti_1 | I0812 18:45:14.917910 8 dispatcher.cpp:78] Adding new service: ExtensionRunnerCore (0x563f5d0360a8) to thread: 139626735974144 (0x563f5cf8bd50) in process 8
ubuntu20-osquery-tuti_1 | I0812 18:45:14.917960 8 tls_enroll.cpp:70] TLSEnrollPlugin requesting a node enroll key from: <https://final.test.company.com/api/v1/osquery/enroll>
ubuntu20-osquery-tuti_1 | I0812 18:45:14.918006 32 interface.cpp:270] Extension manager service starting: /var/osquery/osquery.em
ubuntu20-osquery-tuti_1 | I0812 18:45:14.921299 8 system.cpp:301] Using host identifier: 6b63e2dd-673e-4590-8b58-e7f90f6b404c
ubuntu20-osquery-tuti_1 | I0812 18:45:14.924006 8 smbios_tables.cpp:252] Could not read SMBIOS memory
ubuntu20-osquery-tuti_1 | I0812 18:45:14.925240 8 tls.cpp:255] TLS/HTTPS POST request to URI: <https://final.test.company.com/api/v1/osquery/enroll>
ubuntu20-osquery-tuti_1 | W0812 18:45:16.044317 8 tls_enroll.cpp:77] Failed enrollment request to <https://final.test.company.com/api/v1/osquery/enroll> (No node key returned from TLS enroll plugin) retrying...
ubuntu20-osquery-tuti_1 | I0812 18:45:17.051679 8 smbios_tables.cpp:252] Could not read SMBIOS memory
ubuntu20-osquery-tuti_1 | I0812 18:45:17.053877 8 tls.cpp:255] TLS/HTTPS POST request to URI: <https://final.test.company.com/api/v1/osquery/enroll>
ubuntu20-osquery-tuti_1 | W0812 18:45:17.933704 8 tls_enroll.cpp:77] Failed enrollment request to <https://final.test.company.com/api/v1/osquery/enroll> (No node key returned from TLS enroll plugin) retrying...
ubuntu20-osquery-tuti_1 | I0812 18:45:21.937801 8 smbios_tables.cpp:252] Could not read SMBIOS memory
ubuntu20-osquery-tuti_1 | I0812 18:45:21.939075 8 tls.cpp:255] TLS/HTTPS POST request to URI: <https://final.test.company.com/api/v1/osquery/enroll>
ubuntu20-osquery-tuti_1 | I0812 18:45:22.675884 8 auto_constructed_tables.cpp:97] Removing stale ATC entries
ubuntu20-osquery-tuti_1 | I0812 18:45:22.676192 8 dispatcher.cpp:78] Adding new service: ConfigRefreshRunner (0x563f5cfe1e78) to thread: 139626744366848 (0x563f5d16cd10) in process 8
ubuntu20-osquery-tuti_1 | I0812 18:45:22.676275 8 tls_enroll.cpp:70] TLSEnrollPlugin requesting a node enroll key from: <https://final.test.company.com/api/v1/osquery/enroll>
ubuntu20-osquery-tuti_1 | I0812 18:45:22.678752 8 smbios_tables.cpp:252] Could not read SMBIOS memory
ubuntu20-osquery-tuti_1 | I0812 18:45:22.680732 8 tls.cpp:255] TLS/HTTPS POST request to URI: <https://final.test.company.com/api/v1/osquery/enroll>
ubuntu20-osquery-tuti_1 | {"enroll_secret":"XXXXXXXXXREDACTEDXXXXXXXXXXXXX","host_identifier":"6b63e2dd-673e-4590-8b58-e7f90f6b404c","platform_type":"9","host_details":{"os_version":{"_id":"20.04","arch":"x86_64","codename":"focal","major":"20","minor":"04","name":"Ubuntu","patch":"0","pid_with_namespace":"0","platform":"ubuntu","platform_like":"debian","version":"20.04.2 LTS (Focal Fossa)"},"osquery_info":{"build_distro":"centos7","build_platform":"1","config_hash":"","config_valid":"0","extensions":"active","instance_id":"6b63e2dd-673e-4590-8b58-e7f90f6b404c","pid":"8","platform_mask":"9","start_time":"1660329914","uuid":"e95441f4-0000-0000-a7df-90f533d605dc","version":"4.8.0","watcher":"1"},"system_info":{"computer_name":"8c2d0614551a","cpu_brand":"Intel(R) Core(TM) i5-8257U CPU @ 1.40GHz\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000","cpu_logical_cores":"4","cpu_physical_cores":"4","cpu_subtype":"142","cpu_type":"x86_64","hardware_model":"","hostname":"8c2d0614551a","local_hostname":"8c2d0614551a","physical_memory":"12562321408","uuid":"e95441f4-0000-0000-a7df-90f533d605dc"}}}
ubuntu20-osquery-tuti_1 |
ubuntu20-osquery-tuti_1 | {"enroll_secret":"XXXXXXXXXREDACTEDXXXXXXXXXXXXX","host_identifier":"6b63e2dd-673e-4590-8b58-e7f90f6b404c","platform_type":"9","host_details":{"os_version":{"_id":"20.04","arch":"x86_64","codename":"focal","major":"20","minor":"04","name":"Ubuntu","patch":"0","pid_with_namespace":"0","platform":"ubuntu","platform_like":"debian","version":"20.04.2 LTS (Focal Fossa)"},"osquery_info":{"build_distro":"centos7","build_platform":"1","config_hash":"","config_valid":"0","extensions":"active","instance_id":"6b63e2dd-673e-4590-8b58-e7f90f6b404c","pid":"8","platform_mask":"9","start_time":"1660329914","uuid":"e95441f4-0000-0000-a7df-90f533d605dc","version":"4.8.0","watcher":"1"},"system_info":{"computer_name":"8c2d0614551a","cpu_brand":"Intel(R) Core(TM) i5-8257U CPU @ 1.40GHz\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000","cpu_logical_cores":"4","cpu_physical_cores":"4","cpu_subtype":"142","cpu_type":"x86_64","hardware_model":"","hostname":"8c2d0614551a","local_hostname":"8c2d0614551a","physical_memory":"12562321408","uuid":"e95441f4-0000-0000-a7df-90f533d605dc"}}}
...
ubuntu20-osquery-tuti_1 | {"enroll_secret":"XXXXXXXXXREDACTEDXXXXXXXXXXXXX","host_identifier":"6b63e2dd-673e-4590-8b58-e7f90f6b404c","platform_type":"9","host_details":{"os_version":{"_id":"20.04","arch":"x86_64","codename":"focal","major":"20","minor":"04","name":"Ubuntu","patch":"0","pid_with_namespace":"0","platform":"ubuntu","platform_like":"debian","version":"20.04.2 LTS (Focal Fossa)"},"osquery_info":{"build_distro":"centos7","build_platform":"1","config_hash":"","config_valid":"0","extensions":"active","instance_id":"6b63e2dd-673e-4590-8b58-e7f90f6b404c","pid":"8","platform_mask":"9","start_time":"1660329914","uuid":"e95441f4-0000-0000-a7df-90f533d605dc","version":"4.8.0","watcher":"1"},"system_info":{"computer_name":"8c2d0614551a","cpu_brand":"Intel(R) Core(TM) i5-8257U CPU @ 1.40GHz\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000","cpu_logical_cores":"4","cpu_physicalW0812 18:45:23.544463 8 tls_enroll.cpp:77] Failed enrollment request to <https://final.test.company.com/api/v1/osquery/enroll> (No node key returned from TLS enroll plugin) retrying...
ubuntu20-osquery-tuti_1 | I0812 18:45:24.556339 8 smbios_tables.cpp:252] Could not read SMBIOS memory
ubuntu20-osquery-tuti_1 | I0812 18:45:24.558943 8 tls.cpp:255] TLS/HTTPS POST request to URI: <https://final.test.company.com/api/v1/osquery/enroll>
ubuntu20-osquery-tuti_1 | W0812 18:45:25.489326 8 tls_enroll.cpp:77] Failed enrollment request to <https://final.test.company.com/api/v1/osquery/enroll> (No node key returned from TLS enroll plugin) retrying...
ubuntu20-osquery-tuti_1 | I0812 18:45:29.495386 8 smbios_tables.cpp:252] Could not read SMBIOS memory
ubuntu20-osquery-tuti_1 | I0812 18:45:29.498567 8 tls.cpp:255] TLS/HTTPS POST request to URI: <https://final.test.company.com/api/v1/osquery/enroll>
ubuntu20-osquery-tuti_1 | I0812 18:45:30.327318 8 tls.cpp:255] TLS/HTTPS POST request to URI: <https://final.test.company.com/api/v1/osquery/config>
ubuntu20-osquery-tuti_1 | I0812 18:45:31.055061 8 tls_enroll.cpp:70] TLSEnrollPlugin requesting a node enroll key from: <https://final.test.company.com/api/v1/osquery/enroll>
ubuntu20-osquery-tuti_1 | I0812 18:45:31.061553 8 smbios_tables.cpp:252] Could not read SMBIOS memory
ubuntu20-osquery-tuti_1 | I0812 18:45:31.063885 8 tls.cpp:255] TLS/HTTPS POST request to URI: <https://final.test.company.com/api/v1/osquery/enroll>
ubuntu20-osquery-tuti_1 | W0812 18:45:31.925765 8 tls_enroll.cpp:77] Failed enrollment request to <https://final.test.company.com/api/v1/osquery/enroll> (No node key returned from TLS enroll plugin) retrying...
ubuntu20-osquery-tuti_1 | I0812 18:45:32.679306 39 config.cpp:1206] Refreshing configuration state
ubuntu20-osquery-tuti_1 | I0812 18:45:32.931587 8 smbios_tables.cpp:252] Could not read SMBIOS memory
ubuntu20-osquery-tuti_1 | I0812 18:45:32.934620 8 tls.cpp:255] TLS/HTTPS POST request to URI: <https://final.test.company.com/api/v1/osquery/enroll>
ubuntu20-osquery-tuti_1 | _cores":"4","cpu_subtype":"142","cpu_type":"x86_64","hardware_model":"","hostname":"8c2d0614551a","local_hostname":"8c2d0614551a","physical_memory":"12562321408","uuid":"e95441f4-0000-0000-a7df-90f533d605dc"}}}
ubuntu20-osquery-tuti_1 |
ubuntu20-osquery-tuti_1 | {"enroll_secret":"XXXXXXXXXREDACTEDXXXXXXXXXXXXX","host_identifier":"6b63e2dd-673e-4590-8b58-e7f90f6b404c","platform_type":"9","host_details":{"os_version":{"_id":"20.04","arch":"x86_64","codename":"focal","major":"20","minor":"04","name":"Ubuntu","patch":"0","pid_with_namespace":"0","platform":"ubuntu","platform_like":"debian","version":"20.04.2 LTS (Focal Fossa)"},"osquery_info":{"build_distro":"centos7","build_platform":"1","config_hash":"","config_valid":"0","extensions":"active","instance_id":"6b63e2dd-673e-4590-8b58-e7f90f6b404c","pid":"8","platform_mask":"9","start_time":"1660329914","uuid":"e95441f4-0000-0000-a7df-90f533d605dc","version":"4.8.0","watcher":"1"},"system_info":{"computer_name":"8c2d0614551a","cpu_brand":"Intel(R) Core(TM) i5-8257U CPU @ 1.40GHz\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000","cpu_logical_cores":"4","cpu_physical_cores":"4","cpu_subtype":"142","cpu_type":"x86_64","hardware_model":"","hostname":"8c2d0614551a","local_hostname":"8c2d0614551a","physical_memory":"12562321408","uuid":"e95441f4-0000-0000-a7df-90f533d605dc"}}}
ubuntu20-osquery-tuti_1 |
ubuntu20-osquery-tuti_1 | {"enroll_secret":"XXXXXXXXXREDACTEDXXXXXXXXXXXXX","host_identifier":"6b63e2dd-673e-4590-8b58-e7f90f6b404c","platform_type":"9","host_details":{"os_version":{"_id":"20.04","arch":"x86_64","codename":"focal","major":"20","minor":"04","name":"Ubuntu","patch":"0","pid_with_namespace":"0","platform":"ubuntu","platform_like":"debian","version":"20.04.2 LTS (Focal Fossa)"},"osquery_info":{"build_distro":"centos7","build_platform":"1","config_hash":"","config_valid":"0","extensions":"active","instance_id":"6b63e2dd-673e-4590-8b58-e7f90f6b404c","pid":"8","platform_mask":"9","start_time":"1660329914","uuid":"e95441f4-0000-0000-a7df-90f533d605dc","version":"4.8.0","watcher":"1"},"system_info":{"computer_name":"8c2d0614551a","cpu_brand":"Intel(R) Core(TM) i5-8257U CPU @ 1.40GHz\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000","cpu_logical_cores":"4","cpu_physical_cores":"4","cpu_subtype":"142","cpu_type":"x86_64","hardware_model":"","hostname":"8c2d0614551a","local_hostname":"8c2d0614551a","physical_memory":"12562321408","uuid":"e95441f4-0000-0000-a7df-90f533d605dc"}}}
ubuntu20-osquery-tuti_1 |
ubuntu20-osquery-tuti_1 | {"node_key":""}
ubuntu20-osquery-tuti_1 |
Kathy Satterlee
08/17/2022, 4:34 PMJL
08/27/2022, 8:55 PM