Hi all I m stucking in a problem My server has wildcard cert

Question

Hi all I m stucking in a problem My server has wildcard certificate and the FQDN is when a try to make a enroll the osquery return `Failed enrollment request to No node key returned from TLS enroll plugin retrying ` The same behavior happens without flag tls server cert whats make me think it s no certificate problem The certificate was issued in ACM ALB When a try to make request via curl and send body everything work s fine and fleet return the node My osquery flag is configured with ``` force=true host identifier=instance verbose=true debug tls dump=true tls server certs= etc osquery fleet crt enroll secret env=ENROLL SECRET enroll tls endpoint= api osquery enroll config plugin=tls config tls endpoint= api osquery config config refresh=10 disable distributed=false distributed plugin=tls distributed interval=10 distributed tls max attempts=3 distributed tls read endpoint= api osquery distributed read distributed tls write endpoint= api osquery distributed write logger plugin=tls logger tls endpoint= api osquery log logger tls period=10 disable carver=false carver start endpoint= api osquery carve begin carver continue endpoint= api osquery carve block carver block size=2000000```

roberto · Answer

hey there from what I can tell that error message is not necessarily related to your TLS settings your host is is having trouble enrolling to the Fleet server some troubleshooting ideas 1 check that the env variable`ENROLL SECRET` is set as you re providing ` enroll secret env=ENROLL SECRET` 2 double check that `ENROLL SECRET` has the correct value 3 Try looking at the `osqueryd` output when you add ` tls dump`

JL · Answer

Hi Roberto I checked and everything seems fine I tried generate package from fleetctl and install deb `dpkg i fleet osquery 1 0 0 amd64 deb` `orbit enroll secret=XXXXXXXX fleet url=` opt orbit bin osqueryd linux stable osqueryd pidfile= opt orbit osquery pid database path= opt orbit osquery db extensions socket= opt orbit orbit osquery em logger path= opt orbit osquery log enroll secret env ENROLL SECRET host identifier=uuid tls hostname= enroll tls endpoint= api v1 osquery enroll config plugin=tls config tls endpoint= api v1 osquery config config refresh=60 disable distributed=false distributed plugin=tls distributed tls max attempts=10 distributed tls read endpoint= api v1 osquery distributed read distributed tls write endpoint= api v1 osquery distributed write logger plugin=tls filesystem logger tls endpoint= api v1 osquery log disable carver=false carver disable function=false carver start endpoint= api v1 osquery carve begin carver continue endpoint= api v1 osquery carve block carver block size=2000000 tls server certs opt orbit certs pem force flagfile opt orbit osquery flags W0811 22 58 52 311698 83 tls enroll cpp 101 Failed enrollment request to No node key returned from TLS enroll plugin retrying W0811 22 58 54 000798 83 tls enroll cpp 101 Failed enrollment request to No node key returned from TLS enroll plugin retrying W0811 22 58 58 710438 83 tls enroll cpp 101 Failed enrollment request to No node key returned from TLS enroll plugin retrying W0811 22 59 08 428079 83 tls enroll cpp 101 Failed enrollment request to No node key returned from TLS enroll plugin retrying W0811 22 59 25 125315 83 tls enroll cpp 101 Failed enrollment request to No node key returned from TLS enroll plugin retrying ^CW0811 22 59 25 588577 83 tls enroll cpp 115 Enrollment attempts interrupted due to a shutdown request I0811 22 59 25 589396 201 interface cpp 137 Registering extension com fleetdm orbit osquery extension v1 65082 version= sdk= Get the same error one thing that s different from orbit and my original osquery flags file is tls endpoint in my osquery flag file I m using api osquery enroll and orbit use different path api v1 osquery enroll even so the error is the same in both cases

JL · Answer

I run orbit with insecure and works I think this isn t a good idea for production environment

JL · Answer

I generate new certificate with full FQDN in ACM My fleet server is behind ALB

roberto · Answer

thanks for all the details I still think this is not directly related to your TLS config the fact that the error happens in the osquery TLS plugin is just a red herring to give you more context this error `No node key returned` only happens after a successful connection to the Fleet server if and only if the server returns an empty node key you can verify my claim by looking at osquery s <https github com osquery osquery blob 3da346d218115556aa5640226f6ee6e000bf0e07 plugins remote enroll tls enroll cpp L162 L164|source code> and this error generally happens with an invalid empty enroll secret gt I run orbit with insecure and works I think this isn t a good idea for production environment indeed ` insecure` is not recommended for production usage as it uses invalid certificates and skips TLS verification can you try running orbit with the exact same parameters you did except for ` insecure` again One theory I have it might take a minute or two for the server to start returning the node key for the host maybe we were exiting the process too early

JL · Answer

`can you try running orbit with the exact same parameters you did except for insecure again ` Yes I receive the same erro `No node key returned from TLS enroll plugin retrying ` I m have ALB in front off my fleet But I generated o self signed certificate for the backend that is behind the ALB Do you think this could be a problem I m thinking run fleet server without SSL I can t think nothing more to debug my problem

JL · Answer

I don t know if a told this before but when I get the request body from osquery log tls dump and make a request using CURL this works fine and return a node key Ex ```curl X POST <https final test company com api v1 osquery enroll> d enroll secret EYhVg host identifier a97a4442 4352 4e11 8cc2 ab253a712e31 platform type 9 host details os version id 20 04 arch x86 64 codename focal major 20 minor 04 name Ubuntu patch 0 pid with namespace 0 platform ubuntu platform like debian version 20 04 4 LTS Focal Fossa osquery info build distro centos7 build platform linux config hash config valid 0 extensions active instance id a97a4442 4352 4e11 8cc2 ab253a712e31 pid 9 platform mask 9 start time 1660264491 uuid db38421f 0000 0000 a10b 185683d8e894 version 5 3 0 watcher 1 system info computer name 9e81b139a599 cpu brand Intel R Core TM i5 8257U CPU 1 40GHz u0000 u0000 u0000 u0000 u0000 u0000 u0000 u0000 cpu logical cores 4 cpu physical cores 4 cpu subtype 142 cpu type x86 64 hardware model hostname 9e81b139a599 local hostname 9e81b139a599 physical memory 12562321408 uuid db38421f 0000 0000 a10b 185683d8e894 ``` And i receive this return ``` node key XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX ```

Lucas Rodriguez · Answer

Hi folks Did you run that `curl` command from the host where Orbit fails to enroll

zwass · Answer

< JL> what response do you see from Fleet in those `tls dump` logs after it sends the enroll request

JL · Answer

`Hi folks Did you run that curl command from the host where Orbit fails to enroll ` < Lucas Rodriguez> Yes

JL · Answer

< zwass> ```ubuntu20 osquery tuti 1 | I0812 18 45 14 844377 1 init cpp 342 osquery initialized version=4 8 0 ubuntu20 osquery tuti 1 | I0812 18 45 14 844758 1 system cpp 374 Writing osqueryd pid 1 to var run osqueryd pidfile ubuntu20 osquery tuti 1 | I0812 18 45 14 845443 1 extensions cpp 453 Could not autoload extensions Cannot open file for reading etc osquery extensions load ubuntu20 osquery tuti 1 | I0812 18 45 14 845963 1 dispatcher cpp 78 Adding new service WatcherRunner 0x5599a55933e8 to thread 140256932865792 0x5599a55f6620 in process 1 ubuntu20 osquery tuti 1 | I0812 18 45 14 847767 7 watcher cpp 593 osqueryd watcher 1 executing worker 8 ubuntu20 osquery tuti 1 | I0812 18 45 14 869949 8 init cpp 339 osquery worker initialized watcher=1 ubuntu20 osquery tuti 1 | I0812 18 45 14 870151 8 dispatcher cpp 78 Adding new service WatcherWatcherRunner 0x563f5cf74038 to thread 139627301418752 0x563f5cf6d810 in process 8 ubuntu20 osquery tuti 1 | I0812 18 45 14 870213 8 rocksdb cpp 132 Opening RocksDB handle var osquery osquery db ubuntu20 osquery tuti 1 | I0812 18 45 14 917805 8 dispatcher cpp 78 Adding new service ExtensionWatcher 0x563f5d1041b8 to thread 139626727581440 0x563f5cf8bd70 in process 8 ubuntu20 osquery tuti 1 | I0812 18 45 14 917910 8 dispatcher cpp 78 Adding new service ExtensionRunnerCore 0x563f5d0360a8 to thread 139626735974144 0x563f5cf8bd50 in process 8 ubuntu20 osquery tuti 1 | I0812 18 45 14 917960 8 tls enroll cpp 70 TLSEnrollPlugin requesting a node enroll key from ubuntu20 osquery tuti 1 | I0812 18 45 14 918006 32 interface cpp 270 Extension manager service starting var osquery osquery em ubuntu20 osquery tuti 1 | I0812 18 45 14 921299 8 system cpp 301 Using host identifier 6b63e2dd 673e 4590 8b58 e7f90f6b404c ubuntu20 osquery tuti 1 | I0812 18 45 14 924006 8 smbios tables cpp 252 Could not read SMBIOS memory ubuntu20 osquery tuti 1 | I0812 18 45 14 925240 8 tls cpp 255 TLS HTTPS POST request to URI ubuntu20 osquery tuti 1 | W0812 18 45 16 044317 8 tls enroll cpp 77 Failed enrollment request to No node key returned from TLS enroll plugin retrying ubuntu20 osquery tuti 1 | I0812 18 45 17 051679 8 smbios tables cpp 252 Could not read SMBIOS memory ubuntu20 osquery tuti 1 | I0812 18 45 17 053877 8 tls cpp 255 TLS HTTPS POST request to URI ubuntu20 osquery tuti 1 | W0812 18 45 17 933704 8 tls enroll cpp 77 Failed enrollment request to No node key returned from TLS enroll plugin retrying ubuntu20 osquery tuti 1 | I0812 18 45 21 937801 8 smbios tables cpp 252 Could not read SMBIOS memory ubuntu20 osquery tuti 1 | I0812 18 45 21 939075 8 tls cpp 255 TLS HTTPS POST request to URI ubuntu20 osquery tuti 1 | I0812 18 45 22 675884 8 auto constructed tables cpp 97 Removing stale ATC entries ubuntu20 osquery tuti 1 | I0812 18 45 22 676192 8 dispatcher cpp 78 Adding new service ConfigRefreshRunner 0x563f5cfe1e78 to thread 139626744366848 0x563f5d16cd10 in process 8 ubuntu20 osquery tuti 1 | I0812 18 45 22 676275 8 tls enroll cpp 70 TLSEnrollPlugin requesting a node enroll key from ubuntu20 osquery tuti 1 | I0812 18 45 22 678752 8 smbios tables cpp 252 Could not read SMBIOS memory ubuntu20 osquery tuti 1 | I0812 18 45 22 680732 8 tls cpp 255 TLS HTTPS POST request to URI ubuntu20 osquery tuti 1 | enroll secret XXXXXXXXXREDACTEDXXXXXXXXXXXXX host identifier 6b63e2dd 673e 4590 8b58 e7f90f6b404c platform type 9 host details os version id 20 04 arch x86 64 codename focal major 20 minor 04 name Ubuntu patch 0 pid with namespace 0 platform ubuntu platform like debian version 20 04 2 LTS Focal Fossa osquery info build distro centos7 build platform 1 config hash config valid 0 extensions active instance id 6b63e2dd 673e 4590 8b58 e7f90f6b404c pid 8 platform mask 9 start time 1660329914 uuid e95441f4 0000 0000 a7df 90f533d605dc version 4 8 0 watcher 1 system info computer name 8c2d0614551a cpu brand Intel R Core TM i5 8257U CPU 1 40GHz u0000 u0000 u0000 u0000 u0000 u0000 u0000 u0000 cpu logical cores 4 cpu physical cores 4 cpu subtype 142 cpu type x86 64 hardware model hostname 8c2d0614551a local hostname 8c2d0614551a physical memory 12562321408 uuid e95441f4 0000 0000 a7df 90f533d605dc ubuntu20 osquery tuti 1 | ubuntu20 osquery tuti 1 | enroll secret XXXXXXXXXREDACTEDXXXXXXXXXXXXX host identifier 6b63e2dd 673e 4590 8b58 e7f90f6b404c platform type 9 host details os version id 20 04 arch x86 64 codename focal major 20 minor 04 name Ubuntu patch 0 pid with namespace 0 platform ubuntu platform like debian version 20 04 2 LTS Focal Fossa osquery info build distro centos7 build platform 1 config hash config valid 0 extensions active instance id 6b63e2dd 673e 4590 8b58 e7f90f6b404c pid 8 platform mask 9 start time 1660329914 uuid e95441f4 0000 0000 a7df 90f533d605dc version 4 8 0 watcher 1 system info computer name 8c2d0614551a cpu brand Intel R Core TM i5 8257U CPU 1 40GHz u0000 u0000 u0000 u0000 u0000 u0000 u0000 u0000 cpu logical cores 4 cpu physical cores 4 cpu subtype 142 cpu type x86 64 hardware model hostname 8c2d0614551a local hostname 8c2d0614551a physical memory 12562321408 uuid e95441f4 0000 0000 a7df 90f533d605dc ubuntu20 osquery tuti 1 | enroll secret XXXXXXXXXREDACTEDXXXXXXXXXXXXX host identifier 6b63e2dd 673e 4590 8b58 e7f90f6b404c platform type 9 host details os version id 20 04 arch x86 64 codename focal major 20 minor 04 name Ubuntu patch 0 pid with namespace 0 platform ubuntu platform like debian version 20 04 2 LTS Focal Fossa osquery info build distro centos7 build platform 1 config hash config valid 0 extensions active instance id 6b63e2dd 673e 4590 8b58 e7f90f6b404c pid 8 platform mask 9 start time 1660329914 uuid e95441f4 0000 0000 a7df 90f533d605dc version 4 8 0 watcher 1 system info computer name 8c2d0614551a cpu brand Intel R Core TM i5 8257U CPU 1 40GHz u0000 u0000 u0000 u0000 u0000 u0000 u0000 u0000 cpu logical cores 4 cpu physicalW0812 18 45 23 544463 8 tls enroll cpp 77 Failed enrollment request to No node key returned from TLS enroll plugin retrying ubuntu20 osquery tuti 1 | I0812 18 45 24 556339 8 smbios tables cpp 252 Could not read SMBIOS memory ubuntu20 osquery tuti 1 | I0812 18 45 24 558943 8 tls cpp 255 TLS HTTPS POST request to URI ubuntu20 osquery tuti 1 | W0812 18 45 25 489326 8 tls enroll cpp 77 Failed enrollment request to No node key returned from TLS enroll plugin retrying ubuntu20 osquery tuti 1 | I0812 18 45 29 495386 8 smbios tables cpp 252 Could not read SMBIOS memory ubuntu20 osquery tuti 1 | I0812 18 45 29 498567 8 tls cpp 255 TLS HTTPS POST request to URI ubuntu20 osquery tuti 1 | I0812 18 45 30 327318 8 tls cpp 255 TLS HTTPS POST request to URI ubuntu20 osquery tuti 1 | I0812 18 45 31 055061 8 tls enroll cpp 70 TLSEnrollPlugin requesting a node enroll key from ubuntu20 osquery tuti 1 | I0812 18 45 31 061553 8 smbios tables cpp 252 Could not read SMBIOS memory ubuntu20 osquery tuti 1 | I0812 18 45 31 063885 8 tls cpp 255 TLS HTTPS POST request to URI ubuntu20 osquery tuti 1 | W0812 18 45 31 925765 8 tls enroll cpp 77 Failed enrollment request to No node key returned from TLS enroll plugin retrying ubuntu20 osquery tuti 1 | I0812 18 45 32 679306 39 config cpp 1206 Refreshing configuration state ubuntu20 osquery tuti 1 | I0812 18 45 32 931587 8 smbios tables cpp 252 Could not read SMBIOS memory ubuntu20 osquery tuti 1 | I0812 18 45 32 934620 8 tls cpp 255 TLS HTTPS POST request to URI ubuntu20 osquery tuti 1 | cores 4 cpu subtype 142 cpu type x86 64 hardware model hostname 8c2d0614551a local hostname 8c2d0614551a physical memory 12562321408 uuid e95441f4 0000 0000 a7df 90f533d605dc ubuntu20 osquery tuti 1 | ubuntu20 osquery tuti 1 | enroll secret XXXXXXXXXREDACTEDXXXXXXXXXXXXX host identifier 6b63e2dd 673e 4590 8b58 e7f90f6b404c platform type 9 host details os version id 20 04 arch x86 64 codename focal major 20 minor 04 name Ubuntu patch 0 pid with namespace 0 platform ubuntu platform like debian version 20 04 2 LTS Focal Fossa osquery info build distro centos7 build platform 1 config hash config valid 0 extensions active instance id 6b63e2dd 673e 4590 8b58 e7f90f6b404c pid 8 platform mask 9 start time 1660329914 uuid e95441f4 0000 0000 a7df 90f533d605dc version 4 8 0 watcher 1 system info computer name 8c2d0614551a cpu brand Intel R Core TM i5 8257U CPU 1 40GHz u0000 u0000 u0000 u0000 u0000 u0000 u0000 u0000 cpu logical cores 4 cpu physical cores 4 cpu subtype 142 cpu type x86 64 hardware model hostname 8c2d0614551a local hostname 8c2d0614551a physical memory 12562321408 uuid e95441f4 0000 0000 a7df 90f533d605dc ubuntu20 osquery tuti 1 | ubuntu20 osquery tuti 1 | enroll secret XXXXXXXXXREDACTEDXXXXXXXXXXXXX host identifier 6b63e2dd 673e 4590 8b58 e7f90f6b404c platform type 9 host details os version id 20 04 arch x86 64 codename focal major 20 minor 04 name Ubuntu patch 0 pid with namespace 0 platform ubuntu platform like debian version 20 04 2 LTS Focal Fossa osquery info build distro centos7 build platform 1 config hash config valid 0 extensions active instance id 6b63e2dd 673e 4590 8b58 e7f90f6b404c pid 8 platform mask 9 start time 1660329914 uuid e95441f4 0000 0000 a7df 90f533d605dc version 4 8 0 watcher 1 system info computer name 8c2d0614551a cpu brand Intel R Core TM i5 8257U CPU 1 40GHz u0000 u0000 u0000 u0000 u0000 u0000 u0000 u0000 cpu logical cores 4 cpu physical cores 4 cpu subtype 142 cpu type x86 64 hardware model hostname 8c2d0614551a local hostname 8c2d0614551a physical memory 12562321408 uuid e95441f4 0000 0000 a7df 90f533d605dc ubuntu20 osquery tuti 1 | ubuntu20 osquery tuti 1 | node key ubuntu20 osquery tuti 1 |```

Kathy Satterlee · Answer

Hey < JL> Sorry about the delay in getting back to you here Are you still having trouble getting that host enrolled

JL · Answer

Hi < Kathy Satterlee> I discovered the problem The fleet was behind alb with aws waf enabled and i was using self signed certificate in fleet server I removed self signed certificate from fleet and the enrollment start working properly returning the node key After that I was getting second error in the endpoint api v1 osquery distributed write that s returning 403 I made troubleshooting and checked payload grpc I can t find the rule and make correct adjustments yet But after i remove waf everything works fine