kyle
02/09/2023, 4:57 AMdb.t3.medium
to db.t4g.large
and still seeing performance issues. i temporarily bumped it up to a db.r5.xlarge
and while the cpu didnt max out, the page still spun. we only have less than 300 hosts, so i am not sure how complex this query that is running is. any ideas on how to trace down what is causing the slowness and how to address?count
api call just will not get a response back. the only log i see is a context cancelled
after i close the tab2023/02/09 05:05:43 http: panic serving 10.6.95.127:56528: runtime error: invalid memory address or nil pointer dereference
goroutine 5284 [running]:
net/http.(*conn).serve.func1()
net/http/server.go:1850 +0xbf
panic({0x1b3cbc0, 0x3022060})
runtime/panic.go:890 +0x262
<http://github.com/fleetdm/fleet/v4/server/datastore/mysql.(*Datastore).LoadHostByOrbitNodeKey(0x11008a4|github.com/fleetdm/fleet/v4/server/datastore/mysql.(*Datastore).LoadHostByOrbitNodeKey(0x11008a4>?, {0x22a4058, 0xc001285980}, {0xc001438680, 0x20})
<http://github.com/fleetdm/fleet/v4/server/datastore/mysql/hosts.go:1299|github.com/fleetdm/fleet/v4/server/datastore/mysql/hosts.go:1299> +0x2ef
<http://github.com/fleetdm/fleet/v4/server/service.(*Service).AuthenticateOrbitHost(0xc0005a7800|github.com/fleetdm/fleet/v4/server/service.(*Service).AuthenticateOrbitHost(0xc0005a7800>, {0x22a4058, 0xc001285980}, {0xc001438680, 0x20})
<http://github.com/fleetdm/fleet/v4/server/service/orbit.go:88|github.com/fleetdm/fleet/v4/server/service/orbit.go:88> +0x85
<http://github.com/fleetdm/fleet/v4/server/service.authenticatedOrbitHost.func1(|github.com/fleetdm/fleet/v4/server/service.authenticatedOrbitHost.func1(>{0x22a4058, 0xc001285980}, {0x1b4dce0, 0xc000588e70})
<http://github.com/fleetdm/fleet/v4/server/service/endpoint_middleware.go:132|github.com/fleetdm/fleet/v4/server/service/endpoint_middleware.go:132> +0xb7
<http://github.com/fleetdm/fleet/v4/server/service.logged.func1({0x22a4058|github.com/fleetdm/fleet/v4/server/service.logged.func1({0x22a4058>, 0xc001285980}, {0x1b4dce0?, 0xc000588e70?})
<http://github.com/fleetdm/fleet/v4/server/service/endpoint_middleware.go:225|github.com/fleetdm/fleet/v4/server/service/endpoint_middleware.go:225> +0x35
<http://github.com/fleetdm/fleet/v4/server/service/middleware/authzcheck.(*Middleware).AuthzCheck.func1.1({0x22a4058|github.com/fleetdm/fleet/v4/server/service/middleware/authzcheck.(*Middleware).AuthzCheck.func1.1({0x22a4058>, 0xc001285920}, {0x1b4dce0, 0xc000588e70})
<http://github.com/fleetdm/fleet/v4/server/service/middleware/authzcheck/authzcheck.go:31|github.com/fleetdm/fleet/v4/server/service/middleware/authzcheck/authzcheck.go:31> +0xa2
<http://github.com/go-kit/kit/transport/http.Server.ServeHTTP({0xc000ae91f0|github.com/go-kit/kit/transport/http.Server.ServeHTTP({0xc000ae91f0>, 0xc000a6cbb8, 0x1f07388, {0xc000ab9b60, 0x3, 0x4}, {0xc000b091a0, 0x4, 0x6}, 0x1f07380, ...}, ...)
<http://github.com/go-kit/kit@v0.12.0/transport/http/server.go:121|github.com/go-kit/kit@v0.12.0/transport/http/server.go:121> +0x35b
<http://github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerRequestSize.func2({0x7fb47c4610f8|github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerRequestSize.func2({0x7fb47c4610f8>?, 0xc001280780?}, 0xc001283000)
<http://github.com/prometheus/client_golang@v1.13.0/prometheus/promhttp/instrument_server.go:245|github.com/prometheus/client_golang@v1.13.0/prometheus/promhttp/instrument_server.go:245> +0x77
net/http.HandlerFunc.ServeHTTP(0x7fb47c4610f8?, {0x7fb47c4610f8?, 0xc001280780?}, 0xc001285530?)
net/http/server.go:2109 +0x2f
<http://github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerResponseSize.func1(|github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerResponseSize.func1(>{0x7fb47c4610f8?, 0xc001280730?}, 0xc001283000)
<http://github.com/prometheus/client_golang@v1.13.0/prometheus/promhttp/instrument_server.go:284|github.com/prometheus/client_golang@v1.13.0/prometheus/promhttp/instrument_server.go:284> +0xc5
net/http.HandlerFunc.ServeHTTP(0x22a1a20?, {0x7fb47c4610f8?, 0xc001280730?}, 0x203000?)
net/http/server.go:2109 +0x2f
<http://github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1(|github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1(>{0x22a1a20?, 0xc00123a7e0?}, 0xc001283000)
<http://github.com/prometheus/client_golang@v1.13.0/prometheus/promhttp/instrument_server.go:142|github.com/prometheus/client_golang@v1.13.0/prometheus/promhttp/instrument_server.go:142> +0xb8
net/http.HandlerFunc.ServeHTTP(0x0?, {0x22a1a20?, 0xc00123a7e0?}, 0x7fb47c5160f8?)
net/http/server.go:2109 +0x2f
<http://github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2(|github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2(>{0x22a1a20, 0xc00123a7e0}, 0xc001283000)
<http://github.com/prometheus/client_golang@v1.13.0/prometheus/promhttp/instrument_server.go:104|github.com/prometheus/client_golang@v1.13.0/prometheus/promhttp/instrument_server.go:104> +0xbf
net/http.HandlerFunc.ServeHTTP(0x22a4058?, {0x22a1a20?, 0xc00123a7e0?}, 0x22884c0?)
net/http/server.go:2109 +0x2f
<http://github.com/fleetdm/fleet/v4/server/service.publicIP.func1(|github.com/fleetdm/fleet/v4/server/service.publicIP.func1(>{0x22a1a20, 0xc00123a7e0}, 0xc001282f00)
<http://github.com/fleetdm/fleet/v4/server/service/handler.go:152|github.com/fleetdm/fleet/v4/server/service/handler.go:152> +0x1ae
net/http.HandlerFunc.ServeHTTP(0xc001282e00?, {0x22a1a20?, 0xc00123a7e0?}, 0xc00175b958?)
net/http/server.go:2109 +0x2f
<http://github.com/gorilla/mux.(*Router).ServeHTTP(0xc000376b40|github.com/gorilla/mux.(*Router).ServeHTTP(0xc000376b40>, {0x22a1a20, 0xc00123a7e0}, 0xc001282c00)
<http://github.com/gorilla/mux@v1.8.0/mux.go:210|github.com/gorilla/mux@v1.8.0/mux.go:210> +0x1cf
net/http.(*ServeMux).ServeHTTP(0x0?, {0x22a1a20, 0xc00123a7e0}, 0xc001282c00)
net/http/server.go:2487 +0x149
<http://github.com/fleetdm/fleet/v4/server/launcher.(*Handler).Handler.func1(|github.com/fleetdm/fleet/v4/server/launcher.(*Handler).Handler.func1(>{0x22a1a20, 0xc00123a7e0}, 0xc001282c00)
<http://github.com/fleetdm/fleet/v4/server/launcher/server.go:54|github.com/fleetdm/fleet/v4/server/launcher/server.go:54> +0x1b9
net/http.HandlerFunc.ServeHTTP(0x0?, {0x22a1a20?, 0xc00123a7e0?}, 0x7204f4?)
net/http/server.go:2109 +0x2f
net/http.serverHandler.ServeHTTP({0x229d340?}, {0x22a1a20, 0xc00123a7e0}, 0xc001282c00)
net/http/server.go:2947 +0x30c
net/http.(*conn).serve(0xc00053e000, {0x22a4058, 0xc000cf4210})
net/http/server.go:1991 +0x607
created by net/http.(*Server).Serve
net/http/server.go:3102 +0x4db
Kathy Satterlee
02/09/2023, 3:35 PMzwass
02/09/2023, 4:25 PMt3.medium
should be enough for 300 hosts I would think.kyle
02/09/2023, 4:51 PMKathy Satterlee
02/09/2023, 5:25 PMkyle
02/09/2023, 9:02 PM/api/latest/fleet/software/count?scope=softwareCount&vulnerable=false
Kathy Satterlee
02/10/2023, 10:45 PMkyle
02/16/2023, 5:12 PMKathy Satterlee
02/16/2023, 5:32 PMfleetctl
?
fleetctl trigger --name vulnerabilities
kyle
02/17/2023, 12:13 AMkyle@MacBook-Pro-3 ~ % fleetctl trigger --name vulnerabilities
[+] Sent request to trigger vulnerabilities schedule
but now see
kyle@MacBook-Pro-3 ~ % fleetctl trigger --name vulnerabilities
[!] Conflicts with current status of vulnerabilities schedule: triggered run started 8m6.413s ago
"unlock" and trigger the job that does the cleanups
?cron_stats
tableKathy Satterlee
02/17/2023, 12:30 AMfleetctl trigger --name cleanups_then_aggregation
kyle
02/17/2023, 12:32 AMvulnerabilities
job also completed, according to cron_stats
tableKathy Satterlee
02/17/2023, 12:34 AMkyle
02/17/2023, 12:34 AMKathy Satterlee
02/17/2023, 12:51 AMkyle
05/12/2023, 7:19 PM