https://github.com/osquery/osquery logo
Title
b

Benjamin Herrenschmidt

08/13/2020, 1:49 AM
Hi Folks ! Is there a specific channel to discuss the aarch64 port ? Is anybody tackling the rebase yet ? (Looking at helping...)
a

alessandrogario

08/13/2020, 12:05 PM
hello @Benjamin Herrenschmidt! I don't think there's a channel specific to the arch
I am currently stuck with BPF, so any help is super appreciated
b

Benjamin Herrenschmidt

08/13/2020, 9:57 PM
Allright, going through the internal (Amazon) approval process to work on this. I'm trying to find the text of the CLA and has failed so far ... do you have a copy ? It looks like I would need to send a PR to get it but I need to get it vetted before I'm allowed to send a PR 🙂
Approval sorted, I've started looking at the rebase, it looks reasonably simple Makefile wrangling, shifting files around etc... so far. Please let me know if theres anything I need to be aware of, I know nothing about osquery at this point 🙂 (I'm a kernel guy but this needed doing so I volunteered)
a

alessandrogario

08/19/2020, 11:01 PM
So you will find out that our build system is weird for 2 major reasons 1. We use weird names; until recently we had to support the Buck build system used by Facebook, and we aimed to have 1:1 structure. Buck supports namespaces, and so the original targets re-used the same names over and over again. Buck is no longer there... we'll fix this 2. We re-import all the dependencies manually, by rewriting their build system in CMake
The goals of rewriting the build system of each lib are: 1. Not having to host pre-built dependencies anywhere 2. Automatically propagate all the compilation flags and settings top to bottom without worrying about outdated things 3. Not having to deal with incompatibilities between pre-built libraries and osquery or between pre-built libs and other pre-built libs 4. Preventing libraries from leaking dependencies against the host system (example: we don't want the curl makefiles to link against the system openssl. we want to be able to use the openssl version we have already imported) 5. Build everything statically 6. Fine tune all the config files generated by auto-tools, so that we can filter out things that are not supposed to be there (example: one script could detect we have a library on the system that is not supposed to be found as per official builds) 7. Make sure we target the correct glibc version for the reference system (Ubuntu 16.04) so that we can the binary can run on any distro >= recent than that 8. Help with reproducible builds
So to mark the port as beta: the branch is rebased on master, deleting the buck build system, then we merge it
b

Benjamin Herrenschmidt

08/19/2020, 11:05 PM
Ok, I'll try to identify the ""buck" builds. I was also thinking of splitting the patch to separate the moving files & CMake wrangling from code changes
for bisectability/review
also I need to check if the list of libs has changed, I might have to add/remove some etc...
a

alessandrogario

08/19/2020, 11:06 PM
To mark the port as stable: all the libraries that are used by AArch64 are updated. This means taking an AArch64 Ubuntu 16.04 VM, running the autogen scripts to generate the config.h files, and then go through them all and make sure that there are no unneeded deps that sneaked in due to things installed in the VM. The config.h files are then committed. The library is already build by CMake so that's already done by the awesome contributor!
b

Benjamin Herrenschmidt

08/19/2020, 11:06 PM
anyway, I''m going to spend quality time on this in the next few days (had somethying else tto finish while I was waiting for legal)
a

alessandrogario

08/19/2020, 11:06 PM
To mark the port as officially supported: we have to find a way to add a CI
b

Benjamin Herrenschmidt

08/19/2020, 11:06 PM
ok
a

alessandrogario

08/19/2020, 11:07 PM
But we can start merging things as soon as we rebase, and then I can start generating releases for the website
b

Benjamin Herrenschmidt

08/19/2020, 11:07 PM
I might try to generate an Amazon Linux 2 RPM as well in Extras
once I sort out some python3 module deps, but all that can wait til I've done the basic stuff
a

alessandrogario

08/19/2020, 11:08 PM
adding support for Amazon Linux 2 should be easy, the only thing to fix is the glibc package name that is added as a dependency
b

Benjamin Herrenschmidt

08/19/2020, 11:08 PM
ok
I''ll dig into all this and get back to you if I have blocking questions, thanks
a

alessandrogario

08/19/2020, 11:09 PM
that's another convenient thing we gain in having so much control (and maintenance burden!) over the deps
b

Benjamin Herrenschmidt

08/19/2020, 11:09 PM
yup, that's going to help 🙂
a

alessandrogario

08/19/2020, 11:09 PM
thanks for looking into this! can't express how happy I am 😄
b

Benjamin Herrenschmidt

08/19/2020, 11:11 PM
🙂