Hi Folks I was running my fleet server on EC2 with self sign

Question

Hi Folks I was running my fleet server on EC2 with self signed certificate Now EC2 instance runs behind a load balancer and uses ACM certificates My hosts are not able to connect to fleet server now Do I have to run the fleet server again With the ACM certificate But where do i get the key from

Zach Zeid · Answer

If you re using ACM you won t get access to the private key unless you are using a private CA in ACM

Zach Zeid · Answer

Typically generating the cert somewhere and storing the private key encrypted in Parameter Store is a good bet you then can import the fullchain into ACM

CptOfEvilMinions · Answer

< koba> does your `osquery flags` file have the following flag ` tls server certs=`

CptOfEvilMinions · Answer

If so your issue probably indicates that the Osquery endpoints can t verify the server certificate because it s a new certificate since you moved the server behind ACM In this case you need to obtain the public certificate for the ACM which can be performed with curl and openssl

koba · Answer

< CptOfEvilMinions> Yes what you said makes sense But when you launch the fleet server you provide it with a cert and key in my case it s a self signed cert and key Will my Fleet server still accept the request coming from a host with public ACM certs Or do I have to spin a fresh Fleet server with ACM certs

koba · Answer

< Zach Zeid> Yes I m using ACM cert now and can t get a private key I want to use ACM mainly because I want to get rid of annoying `not secure` warning from browser So if I choose ACM what sort of changes do I have to make to my current environment I m wondering Do I have to launch fresh instance of Fleet server with ACM certs Since the current Fleet server that s running is using my self signed certs and key Do I just start enrolling new hosts with public ACM certs Without having to relaunching my Fleet server gt Typically generating the cert somewhere and storing the private key encrypted in Parameter Store is a good bet you then can import the fullchain into ACM So I can simply import my existing self signed cert into ACM and key in Parameter Store I guess this mean my LB will start service my self singed certs But will I get rid or `not secure` browser warning If not it would not be any different than simply not using ACM at all at least for my use case

CptOfEvilMinions · Answer

< koba> if I understand your setup correctly it looks like `osquery endpoint TLS gt AWS LB with ACM cert TLS gt Kolide Fleet` If this is correct Osquery is probably failing because it can t verify the ACM certificate being provided by the LB which is the endpoint Osquery is connecting too To ensure Osquery is verifying the correct certificate please get the value of your ` tls server certs=` flag from `osquery flags` Next `cat lt filepath of value for tls server certs gt ` output should be PEM formatted public cert Next run `openssl s client connect lt kolide FQDN gt lt port gt showcerts lt dev null 2 gt dev null|openssl x509 outform PEM` The output from both commands should be the identical If they are not please make the necessary changes on the endpoint side Restart Osquery If this doesn t resolve the issue I have some ideas but try this first

Zach Zeid · Answer

for 2 do not do this you re giving hosts the server cert and private key unless you have a strong story around cert management just use the enroll secret for 1 you ll have to run `fleet serve` with the server certs configured so probably depending on your setup

koba · Answer

< CptOfEvilMinions> gt if I understand your setup correctly That s correct But a little backstory to establish that certificate part is more clear I had a perfectly working Fleet Server running with self signed certs and I was able to connect my hosts too without any issues Only problem was I wanted to get rid of browser warning So I put my Fleet Server behind LB and the warning went away Of Course the Fleet web console now started serving new ACM cert was able to see the new ACM cert in browser `view site info` and when hosts started to fail at enrolment it kind of made sense why But since I did not know how to fix this I removed LB from picture I m now back to old `osquery endpoint gt TLS gt Kolide Fleet` setup

koba · Answer

Can I chose to go with ACM if yes What soft of changes do I have to make to Fleet Server

Zach Zeid · Answer

what kind of LB are you using I used LE to generate a cert imported it into ACM and am using that for our NLB

CptOfEvilMinions · Answer

< koba> did you run the commands I specified above

Zach Zeid · Answer

then on fleet I run `fleet serve` with the server cert filepath as a commandline switch docker on ECS

koba · Answer

gt < koba> did you run the commands I specified above yes I did The output is not same The second command says ```unable to load certificate 4367269548 error 09FFF06C PEM routines CRYPTO internal no start line AppleInternal BuildRoot Library Caches com apple xbs Sources libressl libressl 56 40 4 libressl 2 8 crypto pem pem lib c 684 Expecting TRUSTED CERTIFICATE```

koba · Answer

note that I am not running Fleet behind LB anymore backstory explained earlier

koba · Answer

gt what kind of LB are you using I used LE to generate a cert imported it into ACM and am using that for our NLB < Zach Zeid> It s an Application LB And the certs are self signed using `openssl`

Zach Zeid · Answer

hm I m using an NLB