Dear team, we have an issue with FleetDM server crashing and it's easily reproduced. We are running 4.41.1. We get this error message:

2024/01/24 07:39:39 http: panic serving X.X.X.X:47746: runtime error: invalid memory address or nil pointer dereference
goroutine 5569605 [running]:
net/http.(*conn).serve.func1()
	net/http/server.go:1868 +0xb9
panic({0x1d0c160?, 0x3629100?})
	runtime/panic.go:920 +0x270
github.com/fleetdm/fleet/v4/server/datastore/mysql.(*Datastore).getConfigEnableDiskEncryption(0x0?, {0x265ba10?, 0xc000fd7710?}, 0xc?)
	github.com/fleetdm/fleet/v4/server/datastore/mysql/app_configs.go:223 +0x37
github.com/fleetdm/fleet/v4/server/datastore/mysql.(*Datastore).applyHostLabelFilters(_, {_, _}, {_, _, _}, _, {_, _}, {{0x0, ...}, ...})
	github.com/fleetdm/fleet/v4/server/datastore/mysql/labels.go:591 +0x655
github.com/fleetdm/fleet/v4/server/datastore/mysql.(*Datastore).ListHostsInLabel(_, {_, _}, {_, _, _}, _, {{0x0, 0x32, {0xc000ff0567, ...}, ...}, ...})
	github.com/fleetdm/fleet/v4/server/datastore/mysql/labels.go:558 +0x2b6
github.com/fleetdm/fleet/v4/server/service.(*Service).ListHostsInLabel(_, {_, _}, _, {{0x0, 0x32, {0xc000ff0567, 0xc}, 0x0, {0x0, ...}, ...}, ...})
	github.com/fleetdm/fleet/v4/server/service/labels.go:291 +0x139
github.com/fleetdm/fleet/v4/server/service.listHostsInLabelEndpoint({0x265ba10, 0xc000fd7710}, {0x1b918e0?, 0xc000f19900}, {0x2689b58?, 0xc000b3c660?})
	github.com/fleetdm/fleet/v4/server/service/labels.go:259 +0x107
github.com/fleetdm/fleet/v4/server/service.(*authEndpointer).makeEndpoint.func1({0x265ba10?, 0xc000fd7710?}, {0x1b918e0?, 0xc000f19900?})
	github.com/fleetdm/fleet/v4/server/service/endpoint_utils.go:583 +0x3a
github.com/fleetdm/fleet/v4/server/service.authenticatedUser.func1({0x265ba10, 0xc000fd7710}, {0x1b918e0, 0xc000f19900})
	github.com/fleetdm/fleet/v4/server/service/endpoint_middleware.go:188 +0xb3
github.com/fleetdm/fleet/v4/server/service.authenticatedUser.logged.func2({0x265ba10, 0xc000fd7710}, {0x1b918e0?, 0xc000f19900?})
	github.com/fleetdm/fleet/v4/server/service/endpoint_middleware.go:223 +0x2f
github.com/fleetdm/fleet/v4/server/service.newServer.newServer.(*Middleware).AuthzCheck.func1.func2({0x265ba10, 0xc000fd6a80}, {0x1b918e0, 0xc000f19900})
	github.com/fleetdm/fleet/v4/server/service/middleware/authzcheck/authzcheck.go:31 +0x7c
github.com/go-kit/kit/transport/http.Server.ServeHTTP({0xc000a58020, 0xc00052f3b0, 0x219dc58, {0xc000a58050, 0x2, 0x2}, {0xc00052f3c8, 0x3, 0x3}, 0x219dc50, ...}, ...)
	github.com/go-kit/kit@v0.12.0/transport/http/server.go:121 +0x32e
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerRequestSize.func2({0x7f03f439e0e0?, 0xc00fd001e0?}, 0xc0015f5700)
	github.com/prometheus/client_golang@v1.13.0/prometheus/promhttp/instrument_server.go:245 +0x71
net/http.HandlerFunc.ServeHTTP(0x7f03f439e0e0?, {0x7f03f439e0e0?, 0xc00fd001e0?}, 0xc000fd62a0?)
	net/http/server.go:2136 +0x29
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerResponseSize.func1({0x7f03f439e0e0?, 0xc00fd00190?}, 0xc0015f5700)
	github.com/prometheus/client_golang@v1.13.0/prometheus/promhttp/instrument_server.go:284 +0xb9
net/http.HandlerFunc.ServeHTTP(0x2655d78?, {0x7f03f439e0e0?, 0xc00fd00190?}, 0x0?)
	net/http/server.go:2136 +0x29
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1({0x2655d78?, 0xc0015cb180?}, 0xc0015f5700)
	github.com/prometheus/client_golang@v1.13.0/prometheus/promhttp/instrument_server.go:142 +0xa8
net/http.HandlerFunc.ServeHTTP(0xc000fd6270?, {0x2655d78?, 0xc0015cb180?}, 0x1f1c0a0?)
	net/http/server.go:2136 +0x29
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2({0x2655d78, 0xc0015cb180}, 0xc0015f5700)
	github.com/prometheus/client_golang@v1.13.0/prometheus/promhttp/instrument_server.go:104 +0xb0
net/http.HandlerFunc.ServeHTTP(0x265ba10?, {0x2655d78?, 0xc0015cb180?}, 0x2639c00?)
	net/http/server.go:2136 +0x29
github.com/fleetdm/fleet/v4/server/service.publicIP.func1({0x2655d78, 0xc0015cb180}, 0xc0015f5600)
	github.com/fleetdm/fleet/v4/server/service/handler.go:165 +0x1ae
net/http.HandlerFunc.ServeHTTP(0xc0015f5500?, {0x2655d78?, 0xc0015cb180?}, 0x46e11b?)
	net/http/server.go:2136 +0x29
github.com/gorilla/mux.(*Router).ServeHTTP(0xc00111a300, {0x2655d78, 0xc0015cb180}, 0xc0015f5400)
	github.com/gorilla/mux@v1.8.0/mux.go:210 +0x1c5
net/http.(*ServeMux).ServeHTTP(0x4717d2?, {0x2655d78, 0xc0015cb180}, 0xc0015f5400)
	net/http/server.go:2514 +0x142
main.createServeCmd.func1.(*Handler).Handler.func29({0x2655d78?, 0xc0015cb180?}, 0x0?)
	github.com/fleetdm/fleet/v4/server/launcher/server.go:54 +0xa5
net/http.HandlerFunc.ServeHTTP(0x6dd05e0?, {0x2655d78?, 0xc0015cb180?}, 0xc001b85b50?)
	net/http/server.go:2136 +0x29
net/http.serverHandler.ServeHTTP({0xc000fd60c0?}, {0x2655d78?, 0xc0015cb180?}, 0x6?)
	net/http/server.go:2938 +0x8e
net/http.(*conn).serve(0xc001e01a70, {0x265ba10, 0xc001106210})
	net/http/server.go:2009 +0x5f4
created by net/http.(*Server).Serve in goroutine 2276
	net/http/server.go:3086 +0x5cb
2024/01/24 07:39:39 http: panic serving 10.3.144.240:47752: runtime error: invalid memory address or nil pointer dereference
goroutine 5569600 [running]:
net/http.(*conn).serve.func1()
	net/http/server.go:1868 +0xb9
panic({0x1d0c160?, 0x3629100?})
	runtime/panic.go:920 +0x270
github.com/fleetdm/fleet/v4/server/datastore/mysql.(*Datastore).getConfigEnableDiskEncryption(0x0?, {0x265ba10?, 0xc000b229c0?}, 0xc?)
	github.com/fleetdm/fleet/v4/server/datastore/mysql/app_configs.go:223 +0x37
github.com/fleetdm/fleet/v4/server/datastore/mysql.(*Datastore).applyHostLabelFilters(_, {_, _}, {_, _, _}, _, {_, _}, {{0x0, ...}, ...})
	github.com/fleetdm/fleet/v4/server/datastore/mysql/labels.go:591 +0x655
github.com/fleetdm/fleet/v4/server/datastore/mysql.(*Datastore).ListHostsInLabel(_, {_, _}, {_, _, _}, _, {{0x0, 0x32, {0xc0007ff1c7, ...}, ...}, ...})
	github.com/fleetdm/fleet/v4/server/datastore/mysql/labels.go:558 +0x2b6
github.com/fleetdm/fleet/v4/server/service.(*Service).ListHostsInLabel(_, {_, _}, _, {{0x0, 0x32, {0xc0007ff1c7, 0xc}, 0x0, {0x0, ...}, ...}, ...})
	github.com/fleetdm/fleet/v4/server/service/labels.go:291 +0x139
github.com/fleetdm/fleet/v4/server/service.listHostsInLabelEndpoint({0x265ba10, 0xc000b229c0}, {0x1b918e0?, 0xc001179b80}, {0x2689b58?, 0xc000b3c660?})
	github.com/fleetdm/fleet/v4/server/service/labels.go:259 +0x107
github.com/fleetdm/fleet/v4/server/service.(*authEndpointer).makeEndpoint.func1({0x265ba10?, 0xc000b229c0?}, {0x1b918e0?, 0xc001179b80?})
	github.com/fleetdm/fleet/v4/server/service/endpoint_utils.go:583 +0x3a
github.com/fleetdm/fleet/v4/server/service.authenticatedUser.func1({0x265ba10, 0xc000b229c0}, {0x1b918e0, 0xc001179b80})
	github.com/fleetdm/fleet/v4/server/service/endpoint_middleware.go:188 +0xb3
github.com/fleetdm/fleet/v4/server/service.authenticatedUser.logged.func2({0x265ba10, 0xc000b229c0}, {0x1b918e0?, 0xc001179b80?})
	github.com/fleetdm/fleet/v4/server/service/endpoint_middleware.go:223 +0x2f
github.com/fleetdm/fleet/v4/server/service.newServer.newServer.(*Middleware).AuthzCheck.func1.func2({0x265ba10, 0xc00103ddd0}, {0x1b918e0, 0xc001179b80})
	github.com/fleetdm/fleet/v4/server/service/middleware/authzcheck/authzcheck.go:31 +0x7c
github.com/go-kit/kit/transport/http.Server.ServeHTTP({0xc000a58020, 0xc00052f3b0, 0x219dc58, {0xc000a58050, 0x2, 0x2}, {0xc00052f3c8, 0x3, 0x3}, 0x219dc50, ...}, ...)
	github.com/go-kit/kit@v0.12.0/transport/http/server.go:121 +0x32e
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerRequestSize.func2({0x7f03f439e0e0?, 0xc0048059f0?}, 0xc001a0b800)
	github.com/prometheus/client_golang@v1.13.0/prometheus/promhttp/instrument_server.go:245 +0x71
net/http.HandlerFunc.ServeHTTP(0x7f03f439e0e0?, {0x7f03f439e0e0?, 0xc0048059f0?}, 0xc00103d920?)
	net/http/server.go:2136 +0x29
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerResponseSize.func1({0x7f03f439e0e0?, 0xc0048059a0?}, 0xc001a0b800)
	github.com/prometheus/client_golang@v1.13.0/prometheus/promhttp/instrument_server.go:284 +0xb9
net/http.HandlerFunc.ServeHTTP(0x2655d78?, {0x7f03f439e0e0?, 0xc0048059a0?}, 0x0?)
	net/http/server.go:2136 +0x29
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1({0x2655d78?, 0xc00189d500?}, 0xc001a0b800)
	github.com/prometheus/client_golang@v1.13.0/prometheus/promhttp/instrument_server.go:142 +0xa8
net/http.HandlerFunc.ServeHTTP(0xc00103d8f0?, {0x2655d78?, 0xc00189d500?}, 0x1f1c0a0?)
	net/http/server.go:2136 +0x29
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2({0x2655d78, 0xc00189d500}, 0xc001a0b800)
	github.com/prometheus/client_golang@v1.13.0/prometheus/promhttp/instrument_server.go:104 +0xb0
net/http.HandlerFunc.ServeHTTP(0x265ba10?, {0x2655d78?, 0xc00189d500?}, 0x2639c00?)
	net/http/server.go:2136 +0x29
github.com/fleetdm/fleet/v4/server/service.publicIP.func1({0x2655d78, 0xc00189d500}, 0xc001a0b700)
	github.com/fleetdm/fleet/v4/server/service/handler.go:165 +0x1ae
net/http.HandlerFunc.ServeHTTP(0xc001a0b600?, {0x2655d78?, 0xc00189d500?}, 0x46e11b?)
	net/http/server.go:2136 +0x29
github.com/gorilla/mux.(*Router).ServeHTTP(0xc00111a300, {0x2655d78, 0xc00189d500}, 0xc001a0b500)
	github.com/gorilla/mux@v1.8.0/mux.go:210 +0x1c5
net/http.(*ServeMux).ServeHTTP(0x4717d2?, {0x2655d78, 0xc00189d500}, 0xc001a0b500)
	net/http/server.go:2514 +0x142
main.createServeCmd.func1.(*Handler).Handler.func29({0x2655d78?, 0xc00189d500?}, 0x0?)
	github.com/fleetdm/fleet/v4/server/launcher/server.go:54 +0xa5
net/http.HandlerFunc.ServeHTTP(0x6dd05e0?, {0x2655d78?, 0xc00189d500?}, 0xc00009cb50?)
	net/http/server.go:2136 +0x29
net/http.serverHandler.ServeHTTP({0xc00103d3b0?}, {0x2655d78?, 0xc00189d500?}, 0x6?)
	net/http/server.go:2938 +0x8e
net/http.(*conn).serve(0xc000c48d80, {0x265ba10, 0xc001106210})
	net/http/server.go:2009 +0x5f4
created by net/http.(*Server).Serve in goroutine 2276
	net/http/server.go:3086 +0x5cb

Hi @Pierre N! Does this happen with all teams, or a specific team?

Hello

Good deal, that means it’s something awry in the team config (it looks like maybe in the disk encryption settings, but that may be a red herring). Is there anything those teams share in common?

Not really, this is our servers, while our corporate devices team works fine. One team is actually empty

Can you share your team configs?

fleetctl get teams —yaml > teams.yml

As well as a debug archive:

fleetctl debug archive

Either of those can contain some sensitive data, so please DM them to me.

I need to go through the email road yes, but thanks for the help!

I’ll take a look at them in the morning.

No rush, I found a workaround for now

What was the workaround? Might help pin down the root cause.

I just look for hosts from the All Teams view all the time 😞

Yeah. That’s less than ideal.

Checking

Hey! just wanted to let you know that I haven't seen that email come through. It may have gotten caught by our filters because of the attachment (I'm pretty sure that happened with a debug archive before). I'm reaching out internally for some help with that.

Thanks for the update Kathy

Ah! It’s support@fleetdm.com

How did I fail my copy/paste? Anyway, resent again

Got it! We’ll take a look shortly.

MySQL version is 8.0.31

Greetings @Pierre N, Thanks much for the troubleshooting and working on getting logs and configurations. Engineering confirmed this is fixed in 4.43.2. They are suggesting an upgrade and seeing if still having trouble. 4.44.0 is in final testing and should be announced soon.

Noted, we will schedule an update to 4.43