Hey all, I’m running Fleet on ECS and I’ve noticed...
# fleetg
George
09/11/2023, 10:24 AMHey all, I’m running Fleet on ECS and I’ve noticed that my policy automations and integrations suddenly go missing. I’m thinking (unproven) this is due to the ephemeral nature of an ECS task. Has anyone else experienced this issue?
z
Zay Hanlon
09/11/2023, 3:03 PM@Benjamin Edwards Any thoughts on this ^ ?
b
Benjamin Edwards
09/11/2023, 3:41 PMPolicy automation and integrations are stored in a stateful manner by the fleet web service into MySQL.
How are you running MySQL?
g
George
09/12/2023, 8:32 AMI’m using RDS, I followed the setup guide here and reference terraform although our’s has considerably diverged since!
George
09/12/2023, 10:52 AMBut the RDS setup has remained pretty similar
b
Benjamin Edwards
09/12/2023, 12:54 PMAlrighty. Not quite sure then. Can you describe specifically the scenario in which you are experiencing data loss?
g
George
09/12/2023, 12:56 PMSo I’ve added a Jira integration, then I enable automations in policies. I’ve done this twice now and both times after an unknown period of time automations are turned off and the integration disappears. My next test will be to kill the ECS task so another spawns and check if the configuration still exists
b
Benjamin Edwards
09/12/2023, 1:01 PMAre you adding this configuration through the web ui?
g
George
09/12/2023, 1:01 PMYeah
George
09/12/2023, 1:02 PMIt definitely saves somewhere because after the first time it disappeared I was checking back a few times
George
09/12/2023, 1:09 PMOk, assumption was incorrect, I killed the ECS task and the configuration still exists. No idea why it’s suddenly disappearing then!
b
Benjamin Edwards
09/12/2023, 1:33 PMIs anyone apply an old config file obtained from fleetctl get config
g
George
09/12/2023, 2:38 PMNo - I’m the only person with that level of access via fleetctl
b
Benjamin Edwards
09/12/2023, 4:04 PMAre you able to reliably reproduce? I’m trying to think of a scenario in which this would occur but having trouble understanding how things are sometimes persisting fine and other times not.
My idea with fleetctl was someone was coming behind you and applying some old configuration effectively wiping out your changes.
What version of fleet server are you running?
g
George
09/13/2023, 9:00 AMI can enable it and wait for it to be removed. We were running 4.22. I’ve just upgraded to 4.37, lets see whether that makes any difference!
5 Views