Is it possible to use the Polylogyx windows extension with fleet/launcher? Or extensions in general? Seems like I'd need to build launcher packages containing the extension/config to start - Would anything need to be done on the server end?

Nothing needs to be done on the server side. If you are using osqueryd you just need to add the flags to connect to the server in addition to the flags for attaching the extension. If you are using Launcher, you'll probably need to do some editing of the flags used to start osquery in order to get the Polylogyx extension attached.