amp amp

Question

< seph> amp amp < theopolis>

seph · Answer

So you think it is git version dependent If it s consistently just augeas does making it not shallow help Does using a release that s within the last 50 commits help

alessandrogario · Answer

Yeah it is if you were to try it on the same git version that we use on the CI it will work

alessandrogario · Answer

The augeas repository is not huge so I would just clone it fully there are some others however that are causing the issue and they are huge

alessandrogario · Answer

I think we have this issue with around ~4 submodules

seph · Answer

Interesting Do you know how far back the commits for the other broken ones are

seph · Answer

Alternately we could just slap something in that complains if the git version is below XXX

alessandrogario · Answer

it is very erratic I don t think we can just filter with a simple `version gt xxx`

alessandrogario · Answer

I think the shallow modules were squashed inside the main commit that contains the third party refactor

alessandrogario · Answer

Yeah it s all in 6481b34e2382ffb5b4c469de3157c63e0ed23ea3

seph · Answer

I was trying to read it last night it looks like the submodules stuff is all cmake we wrote

seph · Answer

Are we linking to specific shas or tags What happens if you add something like `cd moduledir amp amp git fetch t`

alessandrogario · Answer

I would 1 Remove the shallow everywhere I m not really a fan of that 2 Add ~ git modules to the cache 3 CMake will automatically get the delta from the remote repo OR 1 Remove the shallow everywhere 2 Add a pre build script Script would do this 1 Fully clone all the remotes inside the cached CI folder 2 Go though all the submodules in osquery and update the remote to point inside the cached submodules 3 git submodule sync recursive 4 Resume normal operations

alessandrogario · Answer

Yeah all the submodules are pointing to an upstream release tag exceptions are bad upstream repositories such as libudev cause it s now systemd and they kind of nuked the version history

alessandrogario · Answer

Some of them got upgraded to a new tagged version

alessandrogario · Answer

The script would be easy to build with `git submodule foreach`

seph · Answer

I dunno I really like using shallow for this sort of thing

seph · Answer

I sorta think you re running into submodules being kinda a mess ISTR they re not super well supported

seph · Answer

Either way though if there s a simple enough reproduce case I d try to play with fetch and keep shallow

seph · Answer

Or think a lot harder and ditch the submodules

alessandrogario · Answer

I can reproduce it 100 of the times on my machine

alessandrogario · Answer

it is really that bad on recent git versions I have 2 23 0 here

seph · Answer

If you re up for it try adding that fetch in I don t think it unshallows

alessandrogario · Answer

It doesn t it needs `git fetch unshallow` there s a submodule command for that i don t remember it

alessandrogario · Answer

sometimes you just restart the cmake configuration git tries again and it works

alessandrogario · Answer

but some modules are stubborn and will never work like google logging

seph · Answer

Weird Is there enough of a reproduce case that I can test this with bare git commands in a test repo I can t easily interate inside the osquery dependency system

alessandrogario · Answer

since this happens at configure time it should be usable anywhere without installing anything but git and cmake

alessandrogario · Answer

this should reproduce the issue `git clone <https github com osquery osquery> amp amp mkdir build amp amp cd build amp amp cmake osquery`

seph · Answer

That s not something I can iterate from I can t quickly tinkering with the git commands in the depths of cmake

seph · Answer

Maybe I ll try though What git version is expected to fail

seph · Answer

version 2 21 0 on macOS works

seph · Answer

works fine on macOS git version 2 23 0

alessandrogario · Answer

Interesting then it s more random than I thought

alessandrogario · Answer

we can maybe ask more info to the other contributors that encountered this issue

seph · Answer

macOS vs linux may matter

sharvil · Answer

works fine on git 2 17 2 on macOS as well

theopolis · Answer

Let s try MITMing our connection to GH for failing versions and or stracing and narrowing this down I m interested in the root cause

seph · Answer

Is there an OS this is likely to fail on

theopolis · Answer

for more context I had tried switching all the URIs to `git ` and adding ` git` and was still able to reproduce I was testing inside of `oss fuzz` s Dockerfile which is ``` FROM ``` and I tried the `git` available from a basic `apt get install` as well as the 2 23 version available here

seph · Answer

Trying to make a test case The docker container `<http gcr io oss fuzz base base builder|gcr io oss fuzz base base builder>` has a cmake that s way too old Ditto ubuntu16 04 Though now I m trying with the cmake in stefano s docker files

theopolis · Answer

you can just download an install a new cmake

seph · Answer

Yes that s what s in stefano s dockerfile

seph · Answer

Anyhow I don t have a good reproduce case yet Entirely because cmake does a bunch of toolchain sanity checks and I don t have a linux build environment for it

alessandrogario · Answer

you can in theory just launch the git command manually

alessandrogario · Answer

`git submodule update init depth=1 libraries cmake source augeas src` should do the trick

seph · Answer

Woo That s a reproduce case

seph · Answer

In `ubuntu 16 04` I can get that easily

seph · Answer

Using the above instruction it fails on ubuntu16 git version 2 7 4 But it works with ubuntu18 git version 2 17 1

seph · Answer

`<http gcr io oss fuzz base base builder|gcr io oss fuzz base base builder>` uses git 2 7

seph · Answer

<https github com boostorg boost issues 245> seems to match both what we re seeing here and my understanding of this error

theopolis · Answer

gt sometimes you just restart the cmake configuration git tries again and it works this is so odd I am seeing this behavior with 2 23 0 too

theopolis · Answer

I think I figured it out this commit breaks the behavior we want merged between 2 21 0 and 2 22 0 rc0 <https github com git git commit 34066f06614d6dbc1955c92ba0c3a17dbfeb0077 diff 7891df559f581cdcc9168363e0f932a1>

seph · Answer

Awesome finding that I don t know if git is developed on github or just mirrored there I don t know if that comment will get seen