trying to enroll a container with osquery on it into Fleet b

Question

trying to enroll a container with osquery on it into Fleet but this error occurs when running `osqueryd` and the container never gets enrolled successfully Any ideas why thinking face btw same error occurs upon running `osqueryi` as well ``` osqueryd verbose tls dump I0426 15 59 05 960875 265 init cpp 357 osquery initialized version=5 2 3 I0426 15 59 05 960927 265 init cpp 364 Using default flagfile etc osquery osquery flags default I0426 15 59 05 979229 265 system cpp 354 Found stale process for osqueryd 157 I0426 15 59 05 979316 265 system cpp 386 Writing osqueryd pid 265 to var run osqueryd pidfile I0426 15 59 05 979460 265 extensions cpp 453 Could not autoload extensions Cannot open file for reading etc osquery extensions load I0426 15 59 05 979648 265 dispatcher cpp 78 Adding new service WatcherRunner 0x55ba6f2c72d8 to thread 140231381698112 0x55ba6f2b4dc0 in process 265 I0426 15 59 05 980381 266 watcher cpp 656 osqueryd watcher 265 executing worker 267 I0426 15 59 05 986223 267 init cpp 354 osquery worker initialized watcher=265 I0426 15 59 05 986275 267 init cpp 364 Using default flagfile etc osquery osquery flags default I0426 15 59 05 986357 267 dispatcher cpp 78 Adding new service WatcherWatcherRunner 0x5567575220d8 to thread 140552803788352 0x55675751a9d0 in process 267 I0426 15 59 05 986425 267 rocksdb cpp 132 Opening RocksDB handle var osquery osquery db I0426 15 59 06 008972 267 dispatcher cpp 78 Adding new service ExtensionWatcher 0x5567575fef38 to thread 140552259495488 0x556757608310 in process 267 I0426 15 59 06 009070 267 dispatcher cpp 78 Adding new service ExtensionRunnerCore 0x5567575fed18 to thread 140552267888192 0x556757529ee0 in process 267 I0426 15 59 06 009122 366 interface cpp 299 Extension manager service starting var osquery osquery em E0426 15 59 06 009130 267 shutdown cpp 79 Cannot activate tls amp amp enroll secret path= etc osquery osquery secret amp amp enroll tls endpoint= api v1 osquery enroll amp amp config tls endpoint= api v1 osquery config amp amp tls hostname= amp amp config refresh=300 amp amp config tls accelerated refresh=300 amp amp config tls max attempts=9999 amp amp distributed tls read endpoint= api v1 osquery distributed read amp amp distributed tls write endpoint= api v1 osquery distributed write amp amp carver start endpoint= api v1 osquery carve begin amp amp carver continue endpoint= api v1 osquery carve block config plugin Unknown registry plugin tls amp amp enroll secret path= etc osquery osquery secret amp amp enroll tls endpoint= api v1 osquery enroll amp amp config tls endpoint= api v1 osquery config amp amp tls hostname= amp amp config refresh=300 amp amp config tls accelerated refresh=300 amp amp config tls max attempts=9999 amp amp distributed tls read endpoint= api v1 osquery distributed read amp amp distributed tls write endpoint= api v1 osquery distributed write amp amp carver start endpoint= api v1 osquery carve begin amp amp carver continue endpoint= api v1 osquery carve block I0426 15 59 06 009331 267 dispatcher cpp 149 Thread 140552803806784 requesting a stop I0426 15 59 06 009389 267 dispatcher cpp 156 Service 0x5567575220d8 has been interrupted I0426 15 59 06 009459 267 dispatcher cpp 156 Service 0x5567575fef38 has been interrupted I0426 15 59 06 009536 267 dispatcher cpp 156 Service 0x5567575fed18 has been interrupted I0426 15 59 06 009622 267 dispatcher cpp 122 Thread 140552803806784 requesting a join I0426 15 59 06 010021 267 dispatcher cpp 140 Service thread 0x556757529ee0 has joined I0426 15 59 06 010056 267 dispatcher cpp 140 Service thread 0x556757608310 has joined I0426 15 59 06 010100 267 dispatcher cpp 140 Service thread 0x55675751a9d0 has joined I0426 15 59 06 010123 267 dispatcher cpp 144 Services and threads have been cleared E0426 15 59 08 981992 266 shutdown cpp 79 Worker returned exit status I0426 15 59 08 983098 265 dispatcher cpp 149 Thread 140231381716544 requesting a stop I0426 15 59 08 983296 265 dispatcher cpp 122 Thread 140231381716544 requesting a join I0426 15 59 08 983441 265 dispatcher cpp 140 Service thread 0x55ba6f2b4dc0 has joined I0426 15 59 08 983824 265 dispatcher cpp 144 Services and threads have been cleared```

pvirani · Answer

It s an Ubuntu container and this is what the Dockerfile looks like ```FROM ubuntu 22 04 RUN apt update amp amp apt get install yes curl wget software properties common Add Osquery Binary Adding GPG settings RUN gpg key url= amp amp gpg keyring path= usr share keyrings segment infra archive keyring gpg amp amp echo n Importing packagecloud gpg key amp amp curl fsSL $ gpg key url | gpg dearmor gt $ gpg keyring path amp amp echo done Install Osquery RUN apt key adv keyserver recv keys 1484120AC4E9F8A1A577AEEE97A80C63C9D8B80B amp amp add apt repository deb arch=amd64 deb main amp amp apt get install osquery RUN rm f etc osquery osquery secret amp amp rm f etc osquery osquery flags amp amp rm f etc osquery osquery flags default RUN echo 28ZojsUf7zpOTQWAOrqSuadZCxWBEX14 gt etc osquery osquery secret Fill osquery flags default with stuff RUN echo config plugin=tls amp amp enroll secret path= etc osquery osquery secret amp amp enroll tls endpoint= api v1 osquery enroll amp amp config tls endpoint= api v1 osquery config amp amp tls hostname= lt our hostname gt amp amp config refresh=300 amp amp config tls accelerated refresh=300 amp amp config tls max attempts=9999 amp amp distributed tls read endpoint= api v1 osquery distributed read amp amp distributed tls write endpoint= api v1 osquery distributed write amp amp carver start endpoint= api v1 osquery carve begin amp amp carver continue endpoint= api v1 osquery carve block gt etc osquery osquery flags default RUN ln s etc osquery osquery flags default etc osquery osquery flags CMD bash ```

zwass · Answer

Ah I think the issue is all the ` amp amp ` in the flagfile

pvirani · Answer

geez good catch Thanks Zach

pvirani · Answer

spoke too soon Same error even after removing the ` amp amp `s

pvirani · Answer

will continue to debug with our SRE folks and report here once I find a solution

pvirani · Answer

in case I haven t mentioned this config works perfectly fine on our EC2 base images It only fails on Docker for some reason

zwass · Answer

Can you just copy the working flagfile into the container in your build

pvirani · Answer

yeah Punted on it for now I was gonna start with a clean default Ubuntu base image and run the commands manually but we mostly have the data we need so decided no point spending much time trying to get it to work just yet I ll keep doing it on the side in evenings if when I have time

zwass · Answer

The same flagfile will almost definitely work assuming the container has network connectivity Happy to help when you get back to it