Title
#core
s

seph

10/03/2019, 12:46 PM
So you think it is git version dependent? If it’s consistently just augeas, does making it not-shallow help? Does using a release that’s within the last 50 commits help?
a

alessandrogario

10/03/2019, 12:47 PM
Yeah, it is; if you were to try it on the same git version that we use on the CI, it will work
12:47 PM
The augeas repository is not huge, so I would just clone it fully; there are some others however that are causing the issue and they are huge
12:48 PM
I think we have this issue with around ~4 submodules
s

seph

10/03/2019, 12:48 PM
Interesting. Do you know how far back the commits for the other broken ones are?
12:49 PM
Alternately, we could just slap something in that complains if the git version is below XXX
a

alessandrogario

10/03/2019, 12:49 PM
it is very erratic, I don't think we can just filter with a simple
version > xxx
12:50 PM
I think the shallow modules were squashed inside the main commit that contains the third-party refactor
12:50 PM
Yeah, it's all in 6481b34e2382ffb5b4c469de3157c63e0ed23ea3
s

seph

10/03/2019, 12:54 PM
I was trying to read it last night — it looks like the submodules stuff is all cmake we wrote?
12:54 PM
Are we linking to specific shas or tags? What happens if you add something like
cd ...moduledir && git fetch -t
a

alessandrogario

10/03/2019, 12:57 PM
I would:1. Remove the shallow everywhere; I'm not really a fan of that 2. Add ~/.git/modules to the cache 3. CMake will automatically get the delta from the remote repo OR 1. Remove the shallow everywhere 2. Add a pre-build script Script would do this:1. Fully clone all the remotes inside the cached CI folder 2. Go though all the submodules in osquery and update the remote to point inside the cached submodules 3. git submodule sync --recursive 4. Resume normal operations
12:58 PM
Yeah, all the submodules are pointing to an upstream release tag; exceptions are bad upstream repositories such as libudev ('cause it's now systemd and they kind of nuked the version history)
12:59 PM
(Some of them got upgraded to a new tagged version)
1:00 PM
The script would be easy to build with
git submodule foreach
s

seph

10/03/2019, 1:00 PM
I dunno… I really like using shallow for this sort of thing.
1:00 PM
I sorta think you’re running into submodules being kinda a mess. ISTR they’re not super well supported
1:01 PM
Either way though, if there’s a simple enough reproduce case, I’d try to play with fetch, and keep shallow.
1:01 PM
Or think a lot harder and ditch the submodules
a

alessandrogario

10/03/2019, 1:03 PM
I can reproduce it 100% of the times on my machine
1:04 PM
it is really that bad on recent git versions (I have 2.23.0 here)
s

seph

10/03/2019, 1:04 PM
If you’re up for it, try adding that fetch in? I don’t think it unshallows
a

alessandrogario

10/03/2019, 1:04 PM
It doesn't, it needs
git fetch --unshallow
(there's a submodule command for that, i don't remember it)
1:05 PM
sometimes you just restart the cmake configuration, git tries again and it works
1:05 PM
but some modules are stubborn and will never work (like google logging)
s

seph

10/03/2019, 1:27 PM
Weird. Is there enough of a reproduce case that I can test this with bare git commands in a test repo? I can’t easily interate inside the osquery dependency system.
a

alessandrogario

10/03/2019, 1:28 PM
since this happens at configure time, it should be usable anywhere without installing anything but git and cmake
1:29 PM
this should reproduce the issue
git clone <https://github.com/osquery/osquery> && mkdir build && cd build && cmake ../osquery
s

seph

10/03/2019, 1:30 PM
That’s not something I can iterate from — I can’t quickly tinkering with the git commands in the depths of cmake
1:30 PM
Maybe I’ll try though. What git version is expected to fail
1:32 PM
version 2.21.0 on macOS works
2:17 PM
works fine on macOS git version 2.23.0
a

alessandrogario

10/03/2019, 2:17 PM
Interesting, then it's more random than I thought
2:18 PM
we can maybe ask more info to the other contributors that encountered this issue
s

seph

10/03/2019, 2:18 PM
macOS vs linux may matter
s

sharvil

10/03/2019, 4:35 PM
works fine on git 2.17.2 on macOS as well
theopolis

theopolis

10/04/2019, 12:09 PM
Let’s try MITMing our connection to GH for failing versions, and/or stracing and narrowing this down. I’m interested in the root cause.
s

seph

10/04/2019, 3:03 PM
Is there an OS this is likely to fail on?
theopolis

theopolis

10/04/2019, 8:04 PM
for more context, I had tried switching all the URIs to
git://
and adding
.git
and was still able to reproduce. I was testing inside of oss-fuzz's Dockerfile, which is:
FROM <http://gcr.io/oss-fuzz-base/base-builder|gcr.io/oss-fuzz-base/base-builder>
and I tried the
git
available from a basic
apt-get install
as well as the 2.23 version available here: https://askubuntu.com/questions/568591/how-do-i-install-the-latest-version-of-git-with-apt/568596
s

seph

10/04/2019, 9:42 PM
Trying to make a test case… The docker container
<http://gcr.io/oss-fuzz-base/base-builder|gcr.io/oss-fuzz-base/base-builder>
has a cmake that’s way too old. Ditto ubuntu16.04. Though now I’m trying with the cmake in stefano’s docker files
theopolis

theopolis

10/04/2019, 9:42 PM
you can just download an install a new cmake
s

seph

10/04/2019, 9:43 PM
Yes, that’s what’s in stefano’s dockerfile
9:46 PM
Anyhow, I don’t have a good reproduce case yet. Entirely because cmake does a bunch of toolchain sanity checks, and I don’t have a linux build environment for it.
a

alessandrogario

10/04/2019, 9:47 PM
you can in theory just launch the git command manually
9:48 PM
git submodule update --init --depth=1 libraries/cmake/source/augeas/src
should do the trick
s

seph

10/05/2019, 2:24 AM
Woo! That’s a reproduce case.
2:24 AM
In
ubuntu:16.04
I can get that easily
2:39 AM
Using the above instruction, it fails on ubuntu16, git version 2.7.4 But it works with ubuntu18, git version 2.17.1
2:39 AM
<http://gcr.io/oss-fuzz-base/base-builder|gcr.io/oss-fuzz-base/base-builder>
uses git 2.7
2:40 AM
https://github.com/boostorg/boost/issues/245 seems to match both what we’re seeing here, and my understanding of this error.
theopolis

theopolis

10/05/2019, 8:15 PM
sometimes you just restart the cmake configuration, git tries again and it works
this is so odd, I am seeing this behavior with 2.23.0 too
10:43 PM
I think I figured it out this commit breaks the behavior we want (merged between 2.21.0 and 2.22.0-rc0) https://github.com/git/git/commit/34066f06614d6dbc1955c92ba0c3a17dbfeb0077#diff-7891df559f581cdcc9168363e0f932a1
s

seph

10/06/2019, 12:54 AM
Awesome finding that! I don’t know if git is developed on github, or just mirrored there — I don’t know if that comment will get seen.