https://github.com/osquery/osquery logo
Powered by
#core
Title
# core
f

farfella

02/09/2020, 7:04 PM
#6190 needs a review and merge-- had to make a change to update with changes to master
t

thor

02/10/2020, 6:43 PM
Considering this is quite the sizeable change, do you mind if this is held off until 4.2.1?
f

farfella

02/10/2020, 7:13 PM
Oh, let’s wait! I apologize, I thought we were releasing 4.2.1 since theo had wanted me to merge/rebase master, haha
t

thor

02/10/2020, 7:26 PM
I'm definitely open to discussing it more. I saw that @seph tagged it as 4.2.1, and considering it's a relatively broad change I'd prefer to push it to another release as we're already changing so much in 4.2.0, but maybe that's not the right way to think about it considering 4.2.0 may be a stable candidate, this fix would be nice to have 🙂 What does the rest of TSC think? @zwass @seph @Stefano Bonicatti @alessandrogario ^^
z

zwass

02/10/2020, 7:43 PM
I'd like to get the release out ASAP, so whatever will enable us to do that quicker. Then let's work on increasing the cadence so we can get other new things out without waiting months.
👍 2
s

Stefano Bonicatti

02/10/2020, 7:53 PM
A bit late but on that note I think that for a CVE having a release with only that fix and nothing else might be better for several reasons. We can focus on the fix and the user supposedly have to retest only the code around the fix.
f

farfella

02/10/2020, 7:57 PM
I agree with @Stefano Bonicatti a x.y.z release where z is patch for CVEs/critical issues would be better for users who do not want new features yet
t

thor

02/10/2020, 7:59 PM
I don't disagree, but so much of the testing needed for this bug was baked into quite a few PRs that shipped, and it somewhat goes against the release cadence/style we've commonly used. I'd say that's a really good discussion we should have at an OH, but given how close we are to already tagging 4.2.0, can this be something we explore for future releases?
To the concept that folks might be less inclined to adopt this release and get the fix -- we already have quite a good slew of folks who don't upgrade to the latest cuts, and are running very far back on releases, the issue tracker is a good testament to this as we see lots of 3.X issues brought up.
👍 1
s

Stefano Bonicatti

02/10/2020, 8:00 PM
ah sure, my comment was meant for the future.
👍 1
z

zwass

02/10/2020, 8:00 PM
I do agree we should do so for security fixes in the future.
s

seph

02/10/2020, 8:00 PM
The milestone tagging should be seen as a loose guideline, not a prescriptive assignment… For 4.2.0 specifically, I move ~everythign to 4.2.1 to clear it for the SSL stuff
z

zwass

02/10/2020, 8:01 PM
It sounds like 4.2.1 should probably be 4.3.0?
a

alessandrogario

02/10/2020, 8:01 PM
@thor I’ve handed the PR to Stefano! I initially thought it was easier to just close my PR and add that missing line to Teddy’s PR, so I marked it as do-not-merge (as I didn’t even test it). Stefano can push to that branch if keep the PR open (and remove the do-not-merge tag)
👍 1
t

thor

02/10/2020, 8:02 PM
Ok yeah, so perhaps we can update the vernacular here (https://github.com/osquery/osquery#vulnerabilities) to reflect that for CVEs we'll strive to have minor tagged releases to highlight such things?
s

seph

02/10/2020, 8:02 PM
I don’t have strong feelings about whether 4.2.1 is called 4.3.0 or not.
a

alessandrogario

02/10/2020, 8:02 PM
I think osquery has seen a single commit fixex for big stuff, but in this case it had dependencies on stuff that got disabled in the past
👍 1
s

seph

02/10/2020, 8:03 PM
We won’t always be able to roll a patch release. Like. we’d already merged too much to cut a 4.1.3. Thus, 4.2.0. I don’t think we should get really angsty about any of.
z

zwass

02/10/2020, 8:04 PM
We could always cut a branch from the latest stable release and cherry-pick only the relevant changes. This is I think what Stefano is suggesting for the future?
s

Stefano Bonicatti

02/10/2020, 8:04 PM
Yes; it's a very special case anyway/hopefully 😛
t

thor

02/10/2020, 8:06 PM
I agree with trying that approach, but I think we'll always inevitably hit similar situations to this one, where the fix may not just have one simple PR to ship. If it's the case we can cherry-pick on top of the most recent stable I think it's a good idea, but I think, as @seph eludes, having flexibility in this will be good.
s

Stefano Bonicatti

02/10/2020, 8:07 PM
But it's not our flexibility I'm concerned about though ^^'.
f

farfella

02/10/2020, 8:10 PM
So, Q: folks using 3.x... what prevents them from upgrading to 4.x? Are there any “state”/persistent files that need to be upgraded?
That is, is there a case where: user upgrades to 4.x from 3.x and osqueryd does not run anymore as it depends on some file that is different between versions, for example
s

Stefano Bonicatti

02/10/2020, 8:21 PM
For persistent stuff there's the configuration and event db
about the reasons I can think of some: 1. Customizations/private forks 2. If it ain't broke, don't fix it 3. Extensions need a couple of changes in the build system and the code
👍 1
s

samuel

02/10/2020, 8:24 PM
i think there's a sizable number of people sitting at 3.3.2-ish since 4.x hasn't really been stable until 4.1.2
s

seph

02/10/2020, 8:24 PM
What wasn’t stable about the 4.0 line?
s

samuel

02/10/2020, 8:25 PM
aws was broken for a bit, and i think others were advised to wait for a bit
s

seph

02/10/2020, 8:26 PM
I don’t remember that. There have totally been unstable releases (oops). but I don’t think 4.1 fixed any major stability issues. People may have biases against .0 releases
I sorta disagree that we should backport and cherry pick fixes. It seems nice in theory, but I don’t think we have the time/people to do it well, and it doesn’t feel like a thing we need. AFAIK we don’t generally make big breaking changes/
I think if people keep feeling that. we should ditch semver.
f

farfella

02/10/2020, 8:27 PM
Agree, let’s not backport
We just need to ensure future where user can always safely upgrade from any old to new
It should be a guarantee
s

seph

02/10/2020, 8:28 PM
I think that’s always a goal. I’m leery if making strong guarantees. Agility is important. And trying toi exhaustively test corner cases is hard
s

samuel

02/10/2020, 8:29 PM
in this case i don't think there should be a backport because there's not really anything which would make sense to backport to
s

seph

02/10/2020, 8:29 PM
I could argue backporting to 3.x or 4.1.x. But I am always in favor of rolling forward.
s

samuel

02/10/2020, 8:30 PM
i don't think anyone who is running 4.x already is going to protest not having a backport
f

farfella

02/10/2020, 8:31 PM
Like we need to get folks on 3.X out of 3.X :D
💯 1
s

samuel

02/10/2020, 8:32 PM
for what we have left on 3.3.2, those will jump straight to this release.... we were basically just waiting for a stable-enough 4.x and then everyone took off for thanksgiving and stuff
s

seph

02/10/2020, 8:32 PM
It’s kinda concerning to me if people think of the 4.1 line as unstable.
s

samuel

02/10/2020, 8:32 PM
right now we have a mix of 3.3.2, a weird build of 4.0.2 and some 4.1.2... nothing stops the update now
i don't think 4.1.2 is unstable, but building 4.x was (is?) difficult on some platforms
s

seph

02/10/2020, 8:34 PM
Ah, yeah. The build constraints I can sympathize with 😞
s

samuel

02/10/2020, 8:34 PM
there were some issues with deps and things which weren't documented yet
a

alessandrogario

02/10/2020, 9:06 PM
@samuel can you show us where it’s hard to build osquery on? maybe there’s something we can do about it
on the latest(ish) Ubuntu, macOS and Windows, it should be fairly easy
s

samuel

02/10/2020, 9:07 PM
its been a while, but in particular it has been hard to build a working nupkg from scratch on windows
a

alessandrogario

02/10/2020, 9:07 PM
if there is a specific platform where it’s troublesome, we can take a look at it
ah i see, it’s packaging
I have to look at how it worked before ’cause I don’t know much about this specific format
s

samuel

02/10/2020, 9:08 PM
its always been rough, but maybe it is better now
like if you had a space in your username it didn't work previously
a

alessandrogario

02/10/2020, 9:09 PM
I think we have solved the space issue on our side, but on Windows when we use MSBuild we are constrained to its own limitations
like the 256 char limits on paths
and there is not much we can do about it
but yeah if you feel like trying out the new build process we can tag along
if there’s any issue with it
about nupkg: unless Stefano added it, it’s not there yet
s

seph

02/10/2020, 9:16 PM
TBH I don’t want to support nupkg. Or brew. Or other things that feel like weird third party targets
a

alessandrogario

02/10/2020, 9:17 PM
It seems like we can just append NUPKG to the list of generators
I agree it would be better to use more standard formats
but if it just requires a single line we can try it out
nupkg is for chocolatey right?
s

seph

02/10/2020, 9:19 PM
yeah
a

alessandrogario

02/10/2020, 9:19 PM
i’d like it more if it didn’t involve powershell scripts 😂
s

seph

02/10/2020, 9:19 PM
That too.
s

samuel

02/10/2020, 9:36 PM
Pretty sure nupkg is officially supported by Microsoft now and chocolatey is now just a third party wrapper
s

Stefano Bonicatti

02/10/2020, 9:41 PM
Yeah, visual studio has the integration, plus 
Install-Package
in powershell
f

farfella

02/10/2020, 9:42 PM
imo, we don’t have to support nupkg or chocolatey— msi is acceptable and widespread. I‘d want to reduce different vectors users can use to install to just one. .msi doesn’t require any other dependencies on the system, and handles upgrades nicely.
For users, they’d want the official/signed msi vs building it from scratch, no?
a

alessandrogario

02/10/2020, 9:44 PM
there may be pre-existing deployment strategies based on nupkg, since it was originally supported
i think it's fine not to provide new additional package formats
👍 1
but it would be nice if nupkg would still work for people who were relying on it
(without necessarily investing all the time on it)
we can test the nupkg format on CMake without making the format official (i.e. we can make it work without publishing the package on the website)
what do you think seph?
t

thor

02/10/2020, 9:46 PM
We did chocolatey originally because that's how FB does package distribution internally. The MSI came later as external folks wanted it, but I don't see any reason we can't move away from nupkg based things as we've changed how internaly distro works, so we're not using the external choco packaegs.
That being said, I do personally enjoy being able to 
choco install -yf osquery
on my home boxes.
f

farfella

02/10/2020, 9:48 PM
Is it set up so choco upgrade works if you’re in 3.X to move up to 4.X?
t

thor

02/10/2020, 9:48 PM
It's supposed to be, but its been some time since I've done that upgrade path.
When I've done testing for package releases, the things we ship upstream, which I build using the powershell scripts in the repo, are tested for upgrades. That being said, going backwards might not always work so well due to database changes.
f

farfella

02/10/2020, 9:50 PM
Yeah, only one way.
2 Views
Powered by