<@U7QP20JQH> &amp;&amp; <@U09M563C7> <https://osqu...
# core
So you think it is git version dependent? If it’s consistently just augeas, does making it not-shallow help? Does using a release that’s within the last 50 commits help?
Yeah, it is; if you were to try it on the same git version that we use on the CI, it will work
The augeas repository is not huge, so I would just clone it fully; there are some others however that are causing the issue and they are huge
I think we have this issue with around ~4 submodules
Interesting. Do you know how far back the commits for the other broken ones are?
Alternately, we could just slap something in that complains if the git version is below XXX
it is very erratic, I don't think we can just filter with a simple
version > xxx
I think the shallow modules were squashed inside the main commit that contains the third-party refactor
Yeah, it's all in 6481b34e2382ffb5b4c469de3157c63e0ed23ea3
I was trying to read it last night — it looks like the submodules stuff is all cmake we wrote?
Are we linking to specific shas or tags? What happens if you add something like
cd ...moduledir && git fetch -t
I would: 1. Remove the shallow everywhere; I'm not really a fan of that 2. Add ~/.git/modules to the cache 3. CMake will automatically get the delta from the remote repo OR 1. Remove the shallow everywhere 2. Add a pre-build script Script would do this: 1. Fully clone all the remotes inside the cached CI folder 2. Go though all the submodules in osquery and update the remote to point inside the cached submodules 3. git submodule sync --recursive 4. Resume normal operations
Yeah, all the submodules are pointing to an upstream release tag; exceptions are bad upstream repositories such as libudev ('cause it's now systemd and they kind of nuked the version history)
(Some of them got upgraded to a new tagged version)
The script would be easy to build with
git submodule foreach
I dunno… I really like using shallow for this sort of thing.
I sorta think you’re running into submodules being kinda a mess. ISTR they’re not super well supported
Either way though, if there’s a simple enough reproduce case, I’d try to play with fetch, and keep shallow.
Or think a lot harder and ditch the submodules
I can reproduce it 100% of the times on my machine
it is really that bad on recent git versions (I have 2.23.0 here)
If you’re up for it, try adding that fetch in? I don’t think it unshallows
It doesn't, it needs
git fetch --unshallow
(there's a submodule command for that, i don't remember it)
sometimes you just restart the cmake configuration, git tries again and it works
but some modules are stubborn and will never work (like google logging)
Weird. Is there enough of a reproduce case that I can test this with bare git commands in a test repo? I can’t easily interate inside the osquery dependency system.
since this happens at configure time, it should be usable anywhere without installing anything but git and cmake
this should reproduce the issue
git clone <https://github.com/osquery/osquery> && mkdir build && cd build && cmake ../osquery
That’s not something I can iterate from — I can’t quickly tinkering with the git commands in the depths of cmake
Maybe I’ll try though. What git version is expected to fail
version 2.21.0 on macOS works
works fine on macOS git version 2.23.0
Interesting, then it's more random than I thought
we can maybe ask more info to the other contributors that encountered this issue
macOS vs linux may matter
works fine on git 2.17.2 on macOS as well
Let’s try MITMing our connection to GH for failing versions, and/or stracing and narrowing this down. I’m interested in the root cause.
Is there an OS this is likely to fail on?
for more context, I had tried switching all the URIs to
and adding
and was still able to reproduce. I was testing inside of `oss-fuzz`'s Dockerfile, which is:
Copy code
FROM <http://gcr.io/oss-fuzz-base/base-builder|gcr.io/oss-fuzz-base/base-builder>
and I tried the
available from a basic
apt-get install
as well as the 2.23 version available here: https://askubuntu.com/questions/568591/how-do-i-install-the-latest-version-of-git-with-apt/568596
Trying to make a test case… The docker container
has a cmake that’s way too old. Ditto ubuntu16.04. Though now I’m trying with the cmake in stefano’s docker files
you can just download an install a new cmake
Yes, that’s what’s in stefano’s dockerfile
Anyhow, I don’t have a good reproduce case yet. Entirely because cmake does a bunch of toolchain sanity checks, and I don’t have a linux build environment for it.
you can in theory just launch the git command manually
git submodule update --init --depth=1 libraries/cmake/source/augeas/src
should do the trick
Woo! That’s a reproduce case.
I can get that easily
Using the above instruction, it fails on ubuntu16, git version 2.7.4 But it works with ubuntu18, git version 2.17.1
uses git 2.7
https://github.com/boostorg/boost/issues/245 seems to match both what we’re seeing here, and my understanding of this error.
sometimes you just restart the cmake configuration, git tries again and it works
this is so odd, I am seeing this behavior with 2.23.0 too
I think I figured it out this commit breaks the behavior we want (merged between 2.21.0 and 2.22.0-rc0) https://github.com/git/git/commit/34066f06614d6dbc1955c92ba0c3a17dbfeb0077#diff-7891df559f581cdcc9168363e0f932a1
👍 1
Awesome finding that! I don’t know if git is developed on github, or just mirrored there — I don’t know if that comment will get seen.