https://github.com/osquery/osquery logo
Powered by
#fleet
Title
# fleet
s

Shaun S

10/11/2021, 11:56 PM
Brand new to fleet and osquery in general. I have a test server up using https://github.com/CptOfEvilMinions/FleetDM-Automation. I'm trying to generate an "orbit" package to install the agent on a few nodes using the instructions at the bottom of: https://github.com/fleetdm/fleet/blob/main/docs/01-Using-Fleet/00-Learn-how-to-use-Fleet.md. I get weird errors though.
Copy code
$ fleetctl package -type pkg
{"level":"debug","path":"/tmp/orbit-package308488698","time":"2021-10-11T23:43:54Z","message":"created temp dir"}
{"level":"debug","error":"stat /tmp/orbit-package308488698/root/var/lib/orbit/bin/osqueryd/macos/stable/osqueryd: no such file or directory","time":"2021-10-11T23:43:55Z","message":"stat file"}
{"level":"debug","path":"/tmp/orbit-package308488698/root/var/lib/orbit/bin/osqueryd/macos/stable/osqueryd","time":"2021-10-11T23:43:58Z","message":"got osqueryd"}
{"level":"debug","error":"stat /tmp/orbit-package308488698/root/var/lib/orbit/bin/orbit/macos/stable/orbit: no such file or directory","time":"2021-10-11T23:43:58Z","message":"stat file"}
{"level":"debug","path":"/tmp/orbit-package308488698/root/var/lib/orbit/bin/orbit/macos/stable/orbit","time":"2021-10-11T23:43:59Z","message":"got orbit"}
build pkg: cpio Payload: wait cpio: exit status 1
Copy code
$ fleetctl package -type deb
{"level":"debug","path":"/tmp/orbit-package219856153","time":"2021-10-11T23:46:16Z","message":"created temp dir"}
{"level":"debug","error":"stat /tmp/orbit-package219856153/root/var/lib/orbit/bin/osqueryd/linux/stable/osqueryd: no such file or directory","time":"2021-10-11T23:46:18Z","message":"stat file"}
initialize updates: failed to get osqueryd: exec new version: : fork/exec /tmp/orbit-package219856153/root/var/lib/orbit/staging/osqueryd: no such file or directory
That makes it look like osquery needs to be separately installed inside of the container image?
z

zwass

10/12/2021, 12:16 AM
Osquery does not need to be installed in the container. 
fleetctl
should be all you need. What is the OS that you are attempting this on?
s

Shaun S

10/12/2021, 12:34 AM
The image from that repo is alpine based
z

zwass

10/12/2021, 12:37 AM
Ah, you are running 
fleetctl
on the Docker image?
s

Shaun S

10/12/2021, 1:15 AM
Yes. To try to generate the packages, I docker exec'd into the running container where fleet is running.
c

Chad

10/12/2021, 2:41 AM
I've had to hack it quiet a bit to get it to work in a docker image. The issue is there are multiple docker images that need to be run to build (atleast the pkg). So what I did was when I run the 
fleetctl
in the docker image, I mounted a named volume and exposed the docker service so that 
fleetctl
can spawn sibling instances. Then you need to modify the heat code to point to the named volume. Also creating temp dir didn't work for some reason so I hardcoded a temp dir to get it to work.
s

Shaun S

10/12/2021, 2:46 AM
Oh wow, so fleetctl actually needs to spin up containers in order to do the build?
What base image did you use?
c

Chad

10/12/2021, 3:03 AM
Only for windows packages I think
Alpine as well
But I haven't looked at the latest version. Last time I was working on this was 2/3 versions ago, so not sure if anything has changed
If you're gonna use alpine you will also need to install xar and patch it as well as bomutils
So these are the two volumes I had to mount. First one is to allow 
fleetctl
docker image access to docker service to spawn sibling instances and the second one is to share build artifacts between the images:
Copy code
"volumes": [
                    {
                        "localPath": "/var/run/docker.sock",
                        "containerPath": "/var/run/docker.sock"
                    },
                    {
                        "localPath": "named_volume",
                        "containerPath": "/tmp/orbit-package"
                    }
s

Shaun S

10/12/2021, 3:22 AM
is named_path just an arbitrary directory you created in the cwd?
c

Chad

10/12/2021, 3:39 AM
Do you mean named_volume? No its local docker volume I created. I didn't specify a directory but docker by default does point it to a local directory
s

Shaun S

10/12/2021, 3:52 AM
oh, I didn't know that was a thing!
z

zwass

10/12/2021, 4:21 PM
@Chad very interesting approach! Thank you for sharing. @Shaun S we have not tested 
fleetctl package
within the Docker container as of yet -- there are so many dependencies and weird platform-specific things that we've been testing with installing 
fleetctl
binary on the host machine (via 
npm install -g fleetctl
or just downloading/building the binary)
s

Shaun S

10/12/2021, 4:22 PM
Ah, this is good to know. I'll set up another test server sans docker to further my testing.
z

zwass

10/12/2021, 4:22 PM
Having Docker around is totally fine. You'll just want to install fleetctl on the host and not within the container.
s

Shaun S

10/12/2021, 4:23 PM
Understood.
On this page it refers to downloading 
fleet.zip
. Do I need both fleetctl_v4.4.1_linux.tar.gz and fleet_v4.4.1_linux.tar.gz?
z

zwass

10/12/2021, 4:34 PM
If you've already got a server set up (that's the 
fleet
binary) you'll just need 
fleetctl
to interact with the Fleet server and build osquery packages.
s

Shaun S

10/12/2021, 4:40 PM
Ah! So I can keep the server side of things in docker and just have fleetctl installed locally which will connect to the server to do whatever it needs to build orbit.
z

zwass

10/12/2021, 4:42 PM
You got it 🙂
Please let us know any issues you run into -- getting all the dependencies right for building packages is tricky and we still have some bugs to fix and places to smooth out the experience.
s

Shaun S

10/12/2021, 4:48 PM
will do.
I confirm that this appears to be working. I successfully built a .deb package.
I tried to build a mac pkg but get a dependency missing error:
Copy code
fleetctl package -type pkg
Is there a list of dependencies some where?
c

Chad

10/13/2021, 3:43 AM
@zwass - if you're seeing a need for dockerising the build process, I am happy to share what we are doing (can create a feature/issue). Personally, I think dockerising the build makes sense because: 1. I don't have to install a bunch of outdated dependencies on my host (looking at you xar) 2. Its easy to include in CI/CD or as a github action
s

Shaun S

10/13/2021, 3:44 AM
@Chad if you do that, I'd be more than happy to beta test and provide feedback
🙌 1
ty 1
c

Chad

10/13/2021, 3:45 AM
@Shaun S what error are you getting?
s

Shaun S

10/13/2021, 3:45 AM
oops, bad copy/paste
Copy code
build pkg: mkbom: exec: "mkbom": executable file not found in $PATH
c

Chad

10/13/2021, 3:51 AM
Yeah think that and xar are the two main dependencies you will need.
Just two save you having to troubleshoot it this is how I am installing it in Alpine :
Copy code
# Install xar from source

RUN git clone --depth=1 \
              --branch=master \
              <https://github.com/mackyle/xar.git> \
  && cd /xar/xar; \
  sed -i '332s/^.*$/AC_CHECK_LIB([crypto], [OPENSSL_init_crypto], , [have_libcrypto="0"])/' <http://configure.ac|configure.ac> \
  && ./autogen.sh --noconfigure \
  && ./configure LDFLAGS=-lfts \
  && make \
  && make install \
  && cd / \
  && rm -rf /xar 

 # Install bomutils from sources

RUN git clone --depth=1 \
              --branch=master \
              <https://github.com/hogliux/bomutils.git> \
  && cd bomutils \
  && make \
  && make install \
  && chmod 755 build/bin/mkbom \
  && cp build/bin/mkbom /usr/local/bin/mkbom \
  && cd / \
  && rm -rf /bomutils
Don't install xar unless you need it. If you do need it then you need to patch it, the sed in my code above.
z

zwass

10/13/2021, 4:50 PM
I'd love to make one Docker container that could build for each platform. There are at least a few obstacles: 1) Lots of dependencies, the container might end up being huge (not necessarily that big a problem, but potentially a pain) 2) From my research, it looks very unlikely it will ever be possible to Notarize mac .pkg from any platform besides macOS. This means there may always have to be a step of taking the generated pkg and doing the Notarization from a macOS system. 3) The only way I've gotten WiX (for building MSIs) to work is with a pretty specific image architecture that results in a pretty large image and may not be compatible with the other dependencies.
128 Views
Powered by