Hello everyone we are trying to deploy `FleetDM` on `AWS` us osquery #fleet

Hello everyone, we are trying to deploy `FleetDM` ...

Valentin Pezon

10/18/2023, 8:36 PM

Hello everyone, we are trying to deploy

FleetDM

AWS

, using Terraform :) We already have a private VPC, cluster, subnets, route table.. where we host our apps, so we started to work with the “`BYO-ECS`” module since we just need the Fleet instance to be deployed on a fargate Here is a sample of our config (see image) We have a problem with the

password_secret_arn

, whenever we put the ARN we get this error in the fargate deployment process :

Copy code

Task stopped at: 2023-10-18T09:04:31.028Z
ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve secret from asm: service call has been retried 5 time(s): failed to fetch secret arn:aws:secretsmanager:eu-west-3:xxxxx:secret:FLEET_PRIMO_MYSQL_PASSWORD-xxxx from secrets manager: RequestCanceled: request context canceled caused by: context deadline exceeded. Please check your task network configuration.

This means our ECS cluster can’t reach the

secretsmanager

services Any idea on how we can solve it using terraform ? We noticed that if we clone the

byo-ecs

terraform repo and add

assign_public_ip = true

under the

network_configuration

of resource

resource "aws_ecs_service" "fleet" {}

, it works well and the fargate task manage to deploy without this error We have another error afterwards but it’s another story haha Any idea on what we can add in our own configuration to make this thing work ? Adding

assign_public_ip = true

in the

byo-ecs

repo was just a quick workaround obviously thanks !

Grant Bilstad

10/18/2023, 8:44 PM

Hey @Valentin Pezon Bringing up with our infra team to see if maybe this could be adjusted better or if needing to submit bug/request to TF. Will try to get back with a solution for you. Thanks much!

Benjamin Edwards

10/18/2023, 9:22 PM

hey @Valentin Pezon this seems weird because ECS should be resolving those secrets defined in the task definition before fargate even bothers launching the container. To be clear it isn't the fleet container image that is fetching that secret, but ECS itself. This might be something you'd want to get in touch with AWS support about. I am not sure why assigning a public IP seems to resolve things for you, as its not the containers security group that is an issue here. Ensure your Task Execution Role for fleet has permissions to that secret ARN as well.

Valentin Pezon

10/19/2023, 7:15 AM

Yeah it fails before the container launch, i think it’s because we are in private vpc and we should find a way to allow our private VPC to call aws secretmanager service, like NAT gateway or specific outbound rules.. we tried that without success tho 😞

Benjamin Edwards

10/19/2023, 12:58 PM

Ah. Yeah you’ll need someway to resolve the APIs Amazon is exposing. You can use private endpoints I believe if you have some requirements on network egress. https://docs.aws.amazon.com/whitepapers/latest/aws-privatelink/what-are-vpc-endpoints.html

Valentin Pezon

10/19/2023, 1:29 PM

Allright we managed to fix it ! We have another strange error, now on the prepare db command .. any idea ?

Copy code

2023/10/19 13:23:28 FAIL 20230315104937_EnsureUniformCollation.go (fixing host_users table: aggregating dupes: Error 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '","), ']') as ids
         FROM host_users
         GROUP BY
           host_id,' at line 3), quitting migration.

Kathy Satterlee

10/19/2023, 2:03 PM

Hi again @Valentin Pezon 🙂 What version of MySQL are you working with?

Valentin Pezon

10/19/2023, 2:05 PM

Mysql version 8.0.30

Kathy Satterlee

10/19/2023, 2:16 PM

Taking a look at the migration.

Kathy Satterlee

10/19/2023, 3:01 PM

Is there a Fleet instance running during the

prepare db

step?

Valentin Pezon

10/19/2023, 3:21 PM

Yeah I think the fargate take kept relaunching the fleet instance

Kathy Satterlee

10/19/2023, 4:04 PM

I think that is our culprit. Taking the target/max instances to 0 should prevent those from spinning up while the db task is running.

Benjamin Edwards

10/19/2023, 4:06 PM

double check that the task running the migrations is the same container image as the task that runs the fleet server, if they are different, they might be expecting different database schemas

Kathy Satterlee

10/19/2023, 4:06 PM

Thanks @Benjamin Edwards!

Valentin Pezon

10/19/2023, 7:28 PM

Yeah we runned the migration from our localhost, directly on the target db, because the fleet terraform

migration-module

ended up throwing an error so we tried this as a workaround This thread is getting confusing ahha, I suggest we wrap up this thread here and we continue on our team slack, I will make a proper message with our exact config and proper question regarding our problems Thanks @Kathy Satterlee & @Benjamin Edwards 🙏

Kathy Satterlee

10/19/2023, 8:05 PM

Sound good!

2 Views

Open in Slack

Previous Next