kyle
02/09/2023, 4:57 AMdb.t3.medium
to db.t4g.large
and still seeing performance issues. i temporarily bumped it up to a db.r5.xlarge
and while the cpu didnt max out, the page still spun. we only have less than 300 hosts, so i am not sure how complex this query that is running is. any ideas on how to trace down what is causing the slowness and how to address?kyle
02/09/2023, 4:58 AMcount
api call just will not get a response back. the only log i see is a context cancelled
after i close the tabkyle
02/09/2023, 5:34 AM2023/02/09 05:05:43 http: panic serving 10.6.95.127:56528: runtime error: invalid memory address or nil pointer dereference
goroutine 5284 [running]:
net/http.(*conn).serve.func1()
net/http/server.go:1850 +0xbf
panic({0x1b3cbc0, 0x3022060})
runtime/panic.go:890 +0x262
<http://github.com/fleetdm/fleet/v4/server/datastore/mysql.(*Datastore).LoadHostByOrbitNodeKey(0x11008a4|github.com/fleetdm/fleet/v4/server/datastore/mysql.(*Datastore).LoadHostByOrbitNodeKey(0x11008a4>?, {0x22a4058, 0xc001285980}, {0xc001438680, 0x20})
<http://github.com/fleetdm/fleet/v4/server/datastore/mysql/hosts.go:1299|github.com/fleetdm/fleet/v4/server/datastore/mysql/hosts.go:1299> +0x2ef
<http://github.com/fleetdm/fleet/v4/server/service.(*Service).AuthenticateOrbitHost(0xc0005a7800|github.com/fleetdm/fleet/v4/server/service.(*Service).AuthenticateOrbitHost(0xc0005a7800>, {0x22a4058, 0xc001285980}, {0xc001438680, 0x20})
<http://github.com/fleetdm/fleet/v4/server/service/orbit.go:88|github.com/fleetdm/fleet/v4/server/service/orbit.go:88> +0x85
<http://github.com/fleetdm/fleet/v4/server/service.authenticatedOrbitHost.func1(|github.com/fleetdm/fleet/v4/server/service.authenticatedOrbitHost.func1(>{0x22a4058, 0xc001285980}, {0x1b4dce0, 0xc000588e70})
<http://github.com/fleetdm/fleet/v4/server/service/endpoint_middleware.go:132|github.com/fleetdm/fleet/v4/server/service/endpoint_middleware.go:132> +0xb7
<http://github.com/fleetdm/fleet/v4/server/service.logged.func1({0x22a4058|github.com/fleetdm/fleet/v4/server/service.logged.func1({0x22a4058>, 0xc001285980}, {0x1b4dce0?, 0xc000588e70?})
<http://github.com/fleetdm/fleet/v4/server/service/endpoint_middleware.go:225|github.com/fleetdm/fleet/v4/server/service/endpoint_middleware.go:225> +0x35
<http://github.com/fleetdm/fleet/v4/server/service/middleware/authzcheck.(*Middleware).AuthzCheck.func1.1({0x22a4058|github.com/fleetdm/fleet/v4/server/service/middleware/authzcheck.(*Middleware).AuthzCheck.func1.1({0x22a4058>, 0xc001285920}, {0x1b4dce0, 0xc000588e70})
<http://github.com/fleetdm/fleet/v4/server/service/middleware/authzcheck/authzcheck.go:31|github.com/fleetdm/fleet/v4/server/service/middleware/authzcheck/authzcheck.go:31> +0xa2
<http://github.com/go-kit/kit/transport/http.Server.ServeHTTP({0xc000ae91f0|github.com/go-kit/kit/transport/http.Server.ServeHTTP({0xc000ae91f0>, 0xc000a6cbb8, 0x1f07388, {0xc000ab9b60, 0x3, 0x4}, {0xc000b091a0, 0x4, 0x6}, 0x1f07380, ...}, ...)
<http://github.com/go-kit/kit@v0.12.0/transport/http/server.go:121|github.com/go-kit/kit@v0.12.0/transport/http/server.go:121> +0x35b
<http://github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerRequestSize.func2({0x7fb47c4610f8|github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerRequestSize.func2({0x7fb47c4610f8>?, 0xc001280780?}, 0xc001283000)
<http://github.com/prometheus/client_golang@v1.13.0/prometheus/promhttp/instrument_server.go:245|github.com/prometheus/client_golang@v1.13.0/prometheus/promhttp/instrument_server.go:245> +0x77
net/http.HandlerFunc.ServeHTTP(0x7fb47c4610f8?, {0x7fb47c4610f8?, 0xc001280780?}, 0xc001285530?)
net/http/server.go:2109 +0x2f
<http://github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerResponseSize.func1(|github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerResponseSize.func1(>{0x7fb47c4610f8?, 0xc001280730?}, 0xc001283000)
<http://github.com/prometheus/client_golang@v1.13.0/prometheus/promhttp/instrument_server.go:284|github.com/prometheus/client_golang@v1.13.0/prometheus/promhttp/instrument_server.go:284> +0xc5
net/http.HandlerFunc.ServeHTTP(0x22a1a20?, {0x7fb47c4610f8?, 0xc001280730?}, 0x203000?)
net/http/server.go:2109 +0x2f
<http://github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1(|github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1(>{0x22a1a20?, 0xc00123a7e0?}, 0xc001283000)
<http://github.com/prometheus/client_golang@v1.13.0/prometheus/promhttp/instrument_server.go:142|github.com/prometheus/client_golang@v1.13.0/prometheus/promhttp/instrument_server.go:142> +0xb8
net/http.HandlerFunc.ServeHTTP(0x0?, {0x22a1a20?, 0xc00123a7e0?}, 0x7fb47c5160f8?)
net/http/server.go:2109 +0x2f
<http://github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2(|github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2(>{0x22a1a20, 0xc00123a7e0}, 0xc001283000)
<http://github.com/prometheus/client_golang@v1.13.0/prometheus/promhttp/instrument_server.go:104|github.com/prometheus/client_golang@v1.13.0/prometheus/promhttp/instrument_server.go:104> +0xbf
net/http.HandlerFunc.ServeHTTP(0x22a4058?, {0x22a1a20?, 0xc00123a7e0?}, 0x22884c0?)
net/http/server.go:2109 +0x2f
<http://github.com/fleetdm/fleet/v4/server/service.publicIP.func1(|github.com/fleetdm/fleet/v4/server/service.publicIP.func1(>{0x22a1a20, 0xc00123a7e0}, 0xc001282f00)
<http://github.com/fleetdm/fleet/v4/server/service/handler.go:152|github.com/fleetdm/fleet/v4/server/service/handler.go:152> +0x1ae
net/http.HandlerFunc.ServeHTTP(0xc001282e00?, {0x22a1a20?, 0xc00123a7e0?}, 0xc00175b958?)
net/http/server.go:2109 +0x2f
<http://github.com/gorilla/mux.(*Router).ServeHTTP(0xc000376b40|github.com/gorilla/mux.(*Router).ServeHTTP(0xc000376b40>, {0x22a1a20, 0xc00123a7e0}, 0xc001282c00)
<http://github.com/gorilla/mux@v1.8.0/mux.go:210|github.com/gorilla/mux@v1.8.0/mux.go:210> +0x1cf
net/http.(*ServeMux).ServeHTTP(0x0?, {0x22a1a20, 0xc00123a7e0}, 0xc001282c00)
net/http/server.go:2487 +0x149
<http://github.com/fleetdm/fleet/v4/server/launcher.(*Handler).Handler.func1(|github.com/fleetdm/fleet/v4/server/launcher.(*Handler).Handler.func1(>{0x22a1a20, 0xc00123a7e0}, 0xc001282c00)
<http://github.com/fleetdm/fleet/v4/server/launcher/server.go:54|github.com/fleetdm/fleet/v4/server/launcher/server.go:54> +0x1b9
net/http.HandlerFunc.ServeHTTP(0x0?, {0x22a1a20?, 0xc00123a7e0?}, 0x7204f4?)
net/http/server.go:2109 +0x2f
net/http.serverHandler.ServeHTTP({0x229d340?}, {0x22a1a20, 0xc00123a7e0}, 0xc001282c00)
net/http/server.go:2947 +0x30c
net/http.(*conn).serve(0xc00053e000, {0x22a4058, 0xc000cf4210})
net/http/server.go:1991 +0x607
created by net/http.(*Server).Serve
net/http/server.go:3102 +0x4db
Kathy Satterlee
02/09/2023, 3:35 PMzwass
t3.medium
should be enough for 300 hosts I would think.kyle
02/09/2023, 4:51 PMkyle
02/09/2023, 4:51 PMKathy Satterlee
02/09/2023, 5:25 PMkyle
02/09/2023, 9:02 PMkyle
02/09/2023, 9:02 PMkyle
02/10/2023, 10:12 PM/api/latest/fleet/software/count?scope=softwareCount&vulnerable=false
Kathy Satterlee
02/10/2023, 10:45 PMKathy Satterlee
02/16/2023, 5:09 PMkyle
02/16/2023, 5:12 PMkyle
02/16/2023, 5:13 PMKathy Satterlee
02/16/2023, 5:32 PMfleetctl
?
fleetctl trigger --name vulnerabilities
Kathy Satterlee
02/16/2023, 5:32 PMKathy Satterlee
02/16/2023, 5:34 PMkyle
02/17/2023, 12:13 AMkyle@MacBook-Pro-3 ~ % fleetctl trigger --name vulnerabilities
[+] Sent request to trigger vulnerabilities schedule
but now see
kyle@MacBook-Pro-3 ~ % fleetctl trigger --name vulnerabilities
[!] Conflicts with current status of vulnerabilities schedule: triggered run started 8m6.413s ago
kyle
02/17/2023, 12:13 AM"unlock" and trigger the job that does the cleanups
?kyle
02/17/2023, 12:14 AMcron_stats
tableKathy Satterlee
02/17/2023, 12:30 AMfleetctl trigger --name cleanups_then_aggregation
kyle
02/17/2023, 12:32 AMkyle
02/17/2023, 12:32 AMvulnerabilities
job also completed, according to cron_stats
tableKathy Satterlee
02/17/2023, 12:34 AMkyle
02/17/2023, 12:34 AMkyle
02/17/2023, 12:35 AMkyle
02/17/2023, 12:35 AMKathy Satterlee
02/17/2023, 12:51 AMkyle
05/12/2023, 7:19 PM