Getting lock timeouts in fleet that resulted in our vanta integration dying (CC @Grant Bilstad):

level=error ts=2024-07-31T21:54:30.741675853Z component=http method=POST uri=/api/v1/osquery/distributed/write took=21.547767072s ip_addr=172.71.155.48 x_for_ip_addr="45.16.98.153, 172.71.155.48" err="insert label query executions ([2024-07-31 21:54:09.532596043 +0000 UTC m=+194529.178355160 18 55 2024-07-31 21:54:09.532596043 +0000 UTC m=+194529.178355160 36 55 2024-07-31 21:54:09.532596043 +0000 UTC m=+194529.178355160 42 55]): context canceled || create transaction: context canceled || selecting app config: context canceled"
level=error ts=2024-07-31T21:54:36.02047213Z component=http method=POST uri=/api/fleet/orbit/device_token took=50.040794864s ip_addr=172.69.214.227 x_for_ip_addr="142.112.166.50, 172.69.214.227" err="internal error: failed to set or update device auth token: upsert host's device auth token: Error 1205 (HY000): Lock wait timeout exceeded; try restarting transaction" uuid=1924fb9b-6552-4a1c-9e08-f7a72066eeee
level=error ts=2024-07-31T21:54:36.020550112Z component=http method=POST uri=/api/fleet/orbit/device_token took=51.109074661s ip_addr=172.70.111.104 x_for_ip_addr="69.64.193.74, 172.70.111.104" err="internal error: failed to set or update device auth token: upsert host's device auth token: Error 1205 (HY000): Lock wait timeout exceeded; try restarting transaction" uuid=ee1020fd-742f-46b8-a280-042623c558ea
level=error ts=2024-07-31T21:54:36.02048143Z component=http method=POST uri=/api/fleet/orbit/device_token took=50.009735224s ip_addr=10.24.144.71 x_for_ip_addr=10.24.144.71 err="internal error: failed to set or update device auth token: upsert host's device auth token: Error 1205 (HY000): Lock wait timeout exceeded; try restarting transaction" uuid=ba3be0d1-9c57-4d12-816f-e438be96c35e

Usually, lock wait timeouts are an indication of duplicate hosts in the environment. Are you seeing any other errors? "Invalid node key" and "osquery host with duplicate identifier has enrolled in Fleet and will overwrite existing host data"

Looks like this may have been our mysql server running low on diskspace!

Let me check on that for you @James

hi @James - it looks like we're seeing some errors with your vanta integration again. are you seeing the same?

Sorry @James! We got some wires mixed up on our side. Your Vanta integration is fine.