https://github.com/osquery/osquery logo
Title
s

sanjaykcse

08/03/2020, 6:50 PM
I am trying to lunch kolide fleet in AWS and following the blog https://medium.com/poka-techblog/kolide-fleet-on-aws-fargate-a-quick-start-guide-b77000025206. the perquisite is domain name and SSL keys . suppose my FQDN is "kolide-fleet-test.com" and pointers how to generate the SSL keys ? Do I nee to create out on some other host machine having the a web server hosted with "kolide-fleet-test.com" and copy to the cloudfromation script ??
s

sean.cavanaugh

08/03/2020, 7:57 PM
I set an instance up this weekend in AWS by following these instructions and it worked out well https://kifarunix.com/install-and-setup-kolide-fleet-on-ubuntu-18-04/
s

sanjaykcse

08/04/2020, 6:10 AM
Yes, this works fine for VM instances/EC2 instances . Not able to set for EC2 instances behind a load balancer. While creating the load balancer for TLS port 8080 , it asks for pem and cert.
m

maxwhite

08/04/2020, 6:11 PM
Hello, I am the author of the aforementioned blog post 😉 (the first one); It dates a little bit, but yes you will need to generate the certificates on a different machine if you want to use the scripts "as is". This is because AWS does not let you have access to the private key of ACM certificates (for good reasons) so this has to rely on a work-around
It will also not work behind an ALB because Kolide uses HTTPS as well gRPC on the same port, and AWS ALBs only work in HTTPS (and not gRPC)
Note that this is an example script, you might need to modify it a bit to suit your needs (and to upgrade to Fleet 3.0.0)
s

sanjaykcse

08/05/2020, 1:11 PM
Hello @maxwhite , Thanks for the inputs. In AWS setup I am using Network load balancer and updating the certificates at the time of creation of load balancer ( TLS at port 8080 ) .I have generated the certificates on another server and copied it in the path . Still the container is not coming up in AWS To debug it further, trying to execute the docker on local setup by passing other. Environment variables. Unfortunately ,  the docker is not coming up on local setup also . It’s throwing below error: Migrations already completed. Nothing to do. level=info ts=2020-08-05T12:39:40.0147164Z component=service method=ListUsers user=none err=null took=978.7µs ts=2020-08-05T12:39:40.0160256Z transport=https address=0.0.0.0:8080 msg=listening ts=2020-08-05T12:39:40.0165883Z terminated="tls: failed to find any PEM data in key input" Any pointers , how to pass the PEM data ?
m

maxwhite

08/05/2020, 2:08 PM
If you check my
Dockerfile
https://github.com/PokaInc/kolide-aws-samples/blob/master/kolide-ecs/docker/Dockerfile I pass the path to the PEM as an ENV variable
And its downloaded before the
fleet serve
command here: https://github.com/PokaInc/kolide-aws-samples/blob/master/kolide-ecs/docker/run.sh
Note that a more modern approach to this would be to use Secrets Manager to store the private key, or download it from a private S3 maybe?
👍 1
s

sanjaykcse

08/05/2020, 4:56 PM
Thanks @maxwhite. I got the local docker instance running on local setup.( put hack by commenting out the AWS cert pulling code in run.sh and copying local generated kolide.crrt and kolide.key to docker. seems with tuning the .yml files would get the AWS instances up and running . Appreciate your reply. It was quite helpful .
m

maxwhite

08/05/2020, 5:09 PM
Glad I could help 🙂
👍 1
s

sanjaykcse

08/08/2020, 6:49 PM
Finally got the infra up using AWS Fargate. For our usecase EC2 is preferred over fargate . After polishing ,will publish the changes . Kolide Fleet on AWS Fargate: a Quick-Start guide is a good reference to start with.
j

john

08/09/2020, 5:18 PM
why do you prefer EC2 over Fargate?
s

sanjaykcse

08/10/2020, 4:55 AM
Hi @john,In our use case aprroax. 200K Nodes would be continuously pushing logs/results to fleet manager . the container instance would be propositional to the number os nodes ( SSL connections) . If I consider the AWS pricing , I believe it make sense to go for EC2 containers . Please correct me if my assumptions are wrong. New to Kolide and AWS. Long time pricing is the only factor we would be prefering EC2 over Fargate.
j

john

08/10/2020, 1:27 PM
Ah, so it’s a scaling related choice; that makes more sense. The choice between EC2 and Fargate based ECS usually stems from IAM integration and management overhead; cycling EC2 nodes, keeping them updated etc is a cost that can be absorbed and still be lower than fargate in certain scaling scenarios indeed