https://github.com/osquery/osquery logo
Join Slack
Powered by
Hi Folks, I was running my fleet server on EC2 wi...
# kolide
k
Hi Folks, I was running my fleet server on EC2 with self signed certificate. Now EC2 instance runs behind a load balancer and uses ACM certificates. My hosts are not able to connect to fleet server now. Do I have to run the fleet server again? With the ACM certificate? But where do i get the key from?
z
If you're using ACM, you won't get access to the private key unless you are using a private CA in ACM.
Typically, generating the cert somewhere and storing the private key (encrypted) in Parameter Store is a good bet. you then can import the fullchain into ACM
c
@koba does your 
osquery.flags
file have the following flag: 
--tls_server_certs=
?
If so, your issue probably indicates that the Osquery endpoints can’t verify the server certificate because it’s a new certificate since you moved the server behind ACM. In this case, you need to obtain the public certificate for the ACM which can be performed with curl and openssl.
k
@CptOfEvilMinions Yes what you said makes sense. But when you launch the fleet server you provide it with a cert and key (in my case it's a self signed cert and key). Will my Fleet server still accept the request coming from a host with public ACM certs? Or do I have to spin a fresh Fleet server with ACM certs?
@Zach Zeid Yes I'm using ACM cert now and can't get a private key. I want to use ACM mainly because I want to get rid of annoying 
not secure
warning from browser. So if I choose ACM what sort of changes do I have to make to my current environment? I'm wondering:- • Do I have to launch fresh instance of Fleet server with ACM certs? Since the current Fleet server that's running is using my self signed certs and key. • Do I just start enrolling new hosts with public ACM certs? Without having to relaunching my Fleet server?
Typically, generating the cert somewhere and storing the private key (encrypted) in Parameter Store is a good bet. you then can import the fullchain into ACM
So I can simply import my existing self signed cert into ACM and key in Parameter Store? I guess this mean my LB will start service my self singed certs? But will I get rid or 
not secure
browser warning? If not, it would not be any different than simply not using ACM at all (at least for my use case).
c
@koba if I understand your setup correctly it looks like 
osquery endpoint ---- TLS ---> AWS LB with ACM cert --- TLS -- > Kolide Fleet
. If this is correct, Osquery is probably failing because it can’t “verify” the ACM certificate being provided by the LB - which is the endpoint Osquery is connecting too. To ensure Osquery is verifying the correct certificate please get the value of your 
--tls_server_certs=
flag from 
osquery.flags
. Next, 
cat <filepath of value for tls_server_certs>
(output should be PEM formatted public cert). Next, run 
openssl s_client -connect <kolide FQDN>:<port> -showcerts </dev/null 2>/dev/null|openssl x509 -outform PEM
. The output from both commands should be the identical. If they are not please make the necessary changes on the endpoint side. Restart Osquery. If this doesn’t resolve the issue I have some ideas but try this first
z
for (2) do not do this, you're giving hosts the server cert and private key, unless you have a strong story around cert management, just use the enroll secret. for (1), you'll have to run 
fleet serve
with the server certs configured, so probably depending on your setup.
k
@CptOfEvilMinions
if I understand your setup correctly
That's correct. But a little backstory to establish that certificate part is more clear. I had a perfectly working Fleet Server running with self signed certs and I was able to connect my hosts too without any issues. Only problem was I wanted to get rid of browser warning. So I put my Fleet Server behind LB and the warning went away. Of Course the Fleet web console now started serving new ACM cert (was able to see the new ACM cert in browser 
view site info
) and when hosts started to fail at enrolment it kind of made sense why. But since I did not know how to fix this, I removed LB from picture. I'm now back to old 
osquery endpoint ---> TLS ---> Kolide Fleet
setup.
Can I chose to go with ACM, if yes...What soft of changes do I have to make to Fleet Server?
z
what kind of LB are you using? I used LE to generate a cert, imported it into ACM, and am using that for our NLB.
c
@koba did you run the commands I specified above?
z
then on fleet, I run 
fleet serve
with the server cert filepath as a commandline switch (docker on ECS)
👍 1
k
@koba did you run the commands I specified above?
yes I did. The output is not same. The second command says:
Copy code
unable to load certificate
4367269548:error:09FFF06C:PEM routines:CRYPTO_internal:no start line:/AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/libressl/libressl-56.40.4/libressl-2.8/crypto/pem/pem_lib.c:684:Expecting: TRUSTED CERTIFICATE
note that I am not running Fleet behind LB anymore. (backstory explained earlier)
what kind of LB are you using? I used LE to generate a cert, imported it into ACM, and am using that for our NLB.
@Zach Zeid It's an Application LB. And the certs are self signed using 
openssl
z
hm, I'm using an NLB
Open in Slack
PreviousNext
Powered by