Dawei Zhang
03/31/2022, 6:32 PMzwass
03/31/2022, 6:33 PMDawei Zhang
03/31/2022, 6:33 PMKathy Satterlee
03/31/2022, 8:01 PMTomas Touceda
03/31/2022, 8:17 PMDawei Zhang
04/01/2022, 4:58 PM{
"component": "http",
"err": "timestamp: 2022-03-31T18:30:26Z: error in query ingestion",
"ingestion-err": "campaign waiting for listener (please retry)",
"ip_addr": "10.124.121.115",
"level": "error",
"method": "POST",
"took": "6.469503ms",
"ts": "2022-03-31T18:30:26.444353614Z",
"uri": "/api/v1/osquery/distributed/write",
"x_for_ip_addr": "10.124.121.115"
}
let me know if you need more infoArtem
04/28/2022, 2:36 PMzwass
04/28/2022, 3:49 PMxhr_send
request, this likely means your load balancer (or something in the network) is blocking websockets.Dawei Zhang
04/28/2022, 4:31 PMArtem
04/28/2022, 7:05 PMApr 28 18:55:58 fleet-01.test.tech fleet[3040986]: {"component":"http","err":"timestamp: 2022-04-28T18:55:58Z: error in query ingestion","ingestion-err":"ingesting query software_linux: update host software: insert software: timestamp: 2022-04-28T18:55:58Z: Error 1213: Deadlock found when trying to get lock; try restarting transaction","ip_addr":"172.12.13.14","level":"error","method":"POST","took":"6.156863664s","ts":"2022-04-28T18:55:58.53477351Z","uri":"/api/v1/osquery/distributed/write","x_for_ip_addr":"172.12.13.14"}
Apr 28 18:55:58 fleet-01.test.tech fleet[3040986]: {"component":"http","err":"timestamp: 2022-04-28T18:55:54Z: error in query ingestion || create transaction: timestamp: 2022-04-28T18:55:58Z: context canceled || save host with id 27: timestamp: 2022-04-28T18:55:58Z: context canceled","ingestion-err":"ingesting query software_linux: update host software: insert software: timestamp: 2022-04-28T18:55:54Z: context canceled","ip_addr":"172.12.13.15","level":"error","method":"POST","took":"19.774983596s","ts":"2022-04-28T18:55:58.898478856Z","uri":"/api/v1/osquery/distributed/write","x_for_ip_addr":"172.12.13.15"}
Ad-hoc and scheduled queries work fine. We also know that this is not load balancer problem (direct connection to fleet from osquery represents same problem). So now we try so locate reason between Redis and MySQLTomas Touceda
04/28/2022, 7:08 PMArtem
04/28/2022, 7:09 PMTomas Touceda
04/28/2022, 7:14 PMArtem
04/28/2022, 7:15 PM