I've noticed release build issues have popped up a few times. Is there any reason we don't just have the CI build it upon pushing a

vX.Y.Z

tag?

@sundsta Technically it's already building them at each commit of a PR or commits that land on master. The main problem was where to store them, although it should be possible to have some artifacts published/attached to a build result (which then expires in 1 month or so). I don't recall though how the access to it looked like, as in if it required to have an Azure account that's in our organization (ToB still handles the CI side). But then also signing is an issue and a WIP.

I would love for CI to build packages and publish them. There are 'broad strokes' happening that gets us closer to this, for example moving the repo and package hosting to an osquery foundation controlled AWS account (vs, previously hosted by Facebook). The next big step would be creating an Azure pipeline to publish to some temporary storage that we can inspect and iterate with. I am not sure if anyone is currently working on this. I would love to invest time since package publishing takes a considerable amount of my time right now.

You can store them in the azure pipelines artifacts, or you could also push them to a S3 bucket. We could also have automatic signing if everyone is OK with having the signing keys and their unlock pass phrases as secrets in the CI.

We could also have automatic signing if everyone is OK with having the signing keys and their unlock pass phrases as secrets in the CI

I am OK with this in practice but I would like to see some examples of how others do it. Do you know of other projects that do something similar?

I don't have the documentation handy, but I can follow up early next week

I think that for signing we were thinking about using a separate repo where only people that have access to the signing key can push

Storage we kinda have a story for now…

I did work with GitHub Actions for (exactly) two projects (not much experience, I know!). if you have a branch somewhere maybe we can help

Poke around https://github.com/directionless/osquery/runs/843860806?check_suite_focus=true

For my understanding, what is ToB?

Trail of Bits!

Trail of Bits (the employer of several people here).

Ah, of course!

So I dunno. How would we setup cpack to do signing? Is it possible to split the build from the packaging? If so, we could shove the artifacts from the cmake build somewhere, and ingest them for a cpack sign?

Well, on Win/Mac signing is two parts. First, sign

osqueryd

, then you have to insert it to the MSI/PKG, and then sign the MSI/PKG. And of course notarize the PKG as well

Sure, packaging is a separate target and a separate CMake file which is in the end optional (just need to remove a line that includes it).

backstory, because it may seem weird we have the CI under ToB: after the Buck refactor we no longer had any CI here in the osquery repo. We made a fork to add back CI support with CMake which eventually landed here upstream

That wouldn't be a problem either. We can leave the packaging we have right now connected to the project, so that for test one can install unsigned packages. That can also output already a tgz which could be uploaded to this other place, where the second repo we were talking about manages eveything

From some playing around in github actions, it’s pretty easy to sling artifacts around. I expect similar

I have this thing that also builds osquery: https://github.com/zeek/zeek-agent/blob/master/.github/workflows/linux.yml

I could easily imagine a the azure builds pushing artifacts to S3 for general consumption, and then a separate job pulling for signing/packaging official things. Likely leaner if it’s all together in one place. But I ran into test issues.

@seph Probably helps that GitHub Actions is a Azure Pipelines clone with small tweaks 🙂

Yes. It’s a fork 🙂

It's missing a few things things... most important to me is the lack of the ability to pull in Actions from internal/private repos within the same GH org unless you do some gross work-around using an access token for a user/service account

The packaging has similar vibes, The old docker repo you can push with the implicit token, but the newer container one requires manual token creation.

Yeah, it's getting better all the time. They added org wide secrets in May which made my life so much easier