demonbhao
09/21/2020, 2:38 AMdemonbhao
09/22/2020, 3:48 AMdemonbhao
09/22/2020, 10:02 AMEsteban
09/22/2020, 2:09 PMasvoboda
09/22/2020, 4:21 PMArtem
09/22/2020, 8:52 PMDan Achin
09/22/2020, 10:48 PMdarren
09/23/2020, 3:30 PM{
"component": "http",
"err": "authentication error: missing node key",
"ts": "2020-09-23T15:18:39.879218643Z"
}
Dan Achin
09/23/2020, 4:03 PMEsteban
09/24/2020, 1:11 PMIain S
09/24/2020, 3:11 PMInvite
struct explicitly marks the token json as un-exportable with -
.
As far as I can tell that means the only way to get the token right now is via the rendered invite email.
We were interested in writing some automation that handles user invitation/setup and wondered if there was another way to get the invite token out.RitaVrataski
09/24/2020, 3:29 PMChris Broome
09/24/2020, 5:03 PMdemonbhao
09/25/2020, 3:06 AMEsteban
09/25/2020, 1:41 PMSteve Kupper
09/25/2020, 10:46 PM10.100.29.126 - - [25/Sep/2020:22:34:00 +0000] "POST /api/v1/osquery/distributed/read HTTP/1.1" 499 0 "-" "osquery/4.4.0" "-" "127.0.0.1:8080" "TLSv1.2/ECDHE-RSA-AES128-GCM-SHA256" "-" 15.555 15.683 -
At the same time, the request to distributed/read from the client looks like
I0925 15:41:36.760367 119746560 tls.cpp:253] TLS/HTTPS POST request to URI: <https://fleet.domain.net:443/api/v1/osquery/distributed/read>
{"node_key":"HN32A+71pXAVPF57U63QIANo45P2J5I+"}
And osquery_status.log on fleet server shows this
{"hostIdentifier":"7AD00D8C-E849-5DE8-B20A-BD35D6F6137E","calendarTime":"Fri Sep 25 22:41:36 2020 UTC","unixTime":"1601073696","severity":"0","filename":"tls.cpp","line":"253","message":"TLS/HTTPS POST request to URI: <https://fleet.domain.net:443/api/v1/osquery/distributed/read>","version":"4.4.0","decorations":{"host_uuid":"7AD00D8C-E849-5DE8-B20A-BD35D6F6137E","hostname":"<http://c02w40vchv2r.domain.com|c02w40vchv2r.domain.com>"}}
Anyone know why this is happening?
thank you!
Steve K.straffin
09/28/2020, 2:58 PMSal
09/29/2020, 10:29 AMlevel=info ts=2020-09-29T05:17:51.328659802Z component=service method=EnrollAgent ip_addr=10.124.237.153:34884 x_for_ip_addr= err="save enroll failed: inserting: Error 1205: Lock wait timeout exceeded; try restarting transaction" took=5m23.090762881s
level=info ts=2020-09-29T05:17:51.46834596Z component=service method=EnrollAgent ip_addr=10.124.237.148:38000 x_for_ip_addr= err="save enroll failed: inserting: Error 1205: Lock wait timeout exceeded; try restarting transaction" took=2m49.583970117s
Seems to increase as time goes on and then cascades into fleet becoming completely unresponsive. Is this a database problem or a database settings issue? It's rather out of the blue that this happen twice in the past two days.Esteban
09/29/2020, 2:21 PMSal
09/29/2020, 5:29 PMSam.P
09/30/2020, 2:35 AMSaif Abulkhair
09/30/2020, 6:32 AMSal
09/30/2020, 3:44 PMDBA team response:
Essentially this is not db server tuning issue, but an app tuning issue. I understand this needs to be related to the vendor, however as you can see from the Humio links inserts & updates are clashing on the 'hosts' table. The correct approach is to minimize these deadlock situations arising in the first place. The engine will detect and roll back 1 transaction - eg in detail:
------------------------
LATEST DETECTED DEADLOCK
------------------------
2020-09-30 09:46:45 0x7de11cce9700
*** (1) TRANSACTION:
TRANSACTION 35762332832, ACTIVE 0 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 3 lock struct(s), heap size 1136, 2 row lock(s)
MySQL thread id 1157846, OS thread handle 139400941860608, query id 14939859382 <http://prod-10-32-7-83.pw1.bcc.somehost.com|prod-10-32-7-83.pw1.bcc.somehost.com> 10.32.7.83 tlsproddb_user updating
UPDATE hosts SET
seen_time = '2020-09-30 13:46:45.647691'
WHERE node_key='6u8qphrXq6FhlWXYZfYWHT2xwBHX+SsN'
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 771 page no 7 n bits 88 index PRIMARY of table `tlsproddb`.`hosts` trx id 35762332832 lock_mode X locks rec but not gap waiting
*** (2) TRANSACTION:
TRANSACTION 35762332830, ACTIVE 0 sec updating or deleting
mysql tables in use 1, locked 1
4 lock struct(s), heap size 1136, 3 row lock(s), undo log entries 1
MySQL thread id 1157730, OS thread handle 138405804414720, query id 14939859380 <http://prod-10-32-1-12.pw1.bcc.somehost.com|prod-10-32-1-12.pw1.bcc.somehost.com> 10.32.1.12 tlsproddb_user update
INSERT INTO hosts (
detail_update_time,
label_update_time,
osquery_host_id,
seen_time,
node_key,
enroll_secret_name
) VALUES ('1970-01-02 00:00:00', '1970-01-02 00:00:00', 'DB938000-FD74-11E7-8000-000000000000', '2020-09-30 13:46:45.647665', 'WqAC/7dPeDzkh90b5yb4TIIvvJrgd4ut', 'default')
ON DUPLICATE KEY UPDATE
node_key = VALUES(node_key),
deleted = FALSE
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 771 page no 7 n bits 88 index PRIMARY of table `tlsproddb`.`hosts` trx id 35762332830 lock_mode X locks rec but not gap
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 771 page no 3489 n bits 248 index idx_host_unique_nodekey of table `tlsproddb`.`hosts` trx id 35762332830 lock_mode X locks rec but not gap waiting
*** WE ROLL BACK TRANSACTION (1)
Any help on this matter would be greatly appreciated.Esteban
09/30/2020, 7:48 PMSam.P
10/01/2020, 6:49 AMIain S
10/01/2020, 2:13 PMservice
library and have put up a PR for a missing DeleteHost
method on the client struct https://github.com/kolide/fleet/pull/2312Steve Kupper
10/01/2020, 3:46 PMOct 01 15:38:10 <http://hostname.domain.net|hostname.domain.net> bash[8725]: {"component":"http","err":"read auth token: reading from websocket: sockjs: session not in open state","msg":"failed to read auth token","ts":"2020-10-01T15:38:10.35797483Z"}
I am also not getting anything in the results logs. Is there something in need to look at to figure this out? We are making a connection through NGINX.
Nginx config is simple, no grpc pass
location / {
proxy_pass <http://127.0.0.1:8080>;
proxy_read_timeout 900;
proxy_connect_timeout 90;
proxy_redirect <http://127.0.0.1:8080>
<https://nginxserver.domain.net>;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Proxy "";
}
Hink
10/01/2020, 3:56 PMJulian Scala
10/01/2020, 9:50 PMmanikant singh
10/02/2020, 12:45 PM