Title
#fleet
r

Rafa

06/30/2022, 1:48 PM
Hello everyone! I am trying to deploy fleet on AWS but I am having problems with the agent. What I did: 1- create a RDS 2- create a Redis 3- create a self-signed certificate 4- create an EC2, download fleet and start it. 5- generate an installer to connect to Fleet But the host is never connected. I checked the connectivity between them and it is fine. So I am with some doubts: 1 - the installer should generate a process in my machine, right? I found nothing running with fleet in the name or osquery 2- is there a debug mode for the connector?
Lucas Rodriguez

Lucas Rodriguez

06/30/2022, 1:53 PM
Hi Rafa!, a couple of questions 1. Which OS did you install the generated package? 2. If the Fleet URL serves a self-signed certificate (not trusted by your system's CA root bundle) then you will need to specify it when generating the package
fleetctl package ... --fleet-certificate=fleet.pem ...
, did you set such flag?
r

Rafa

06/30/2022, 1:57 PM
Hi Lucas! Fleet is running in Ubuntu 22.04 and the generated package in Ubuntu 20.04.4. I generate the package in the machine that I want to connect with: fleetctl package --type=deb --fleet-url=https://ip:8080 --enroll-secret=ASDASDASDASDASDASDASDASD --fleet-certificate=fleet_osquery.pem LHi
Lucas Rodriguez

Lucas Rodriguez

06/30/2022, 1:58 PM
OK, once the package is generated, did you install it via
sudo dpkg --install
?
r

Rafa

06/30/2022, 1:58 PM
(I omit the true ip and enroll-secret)
1:59 PM
Yes, running sudo dpkg -i fleet-osquery_0.0.13_amd64.deb
Lucas Rodriguez

Lucas Rodriguez

06/30/2022, 1:59 PM
OK, let's check logs then:
1:59 PM
sudo vim /var/log/syslog
(vim or other text editor, and look for orbit/osquery logs)
r

Rafa

06/30/2022, 2:02 PM
Good! Found something:
2:02 PM
Jun 30 10:53:30 orbit[3648258]: W0630 10:53:30.888624 3648258 tls_enroll.cpp:101] Failed enrollment request to https://ip:8080/api/v1/osquery/enroll (Request error: certificate verify failed) retrying...
Lucas Rodriguez

Lucas Rodriguez

06/30/2022, 2:03 PM
OK, osquery is complaining about the self-signed certificate (doesn't trust it).
r

Rafa

06/30/2022, 2:04 PM
But setting --fleet-certificate should solve this right?
Lucas Rodriguez

Lucas Rodriguez

06/30/2022, 2:04 PM
Did you set
--fleet-certificate=fleet_osquery.pem
and it still doesn't work?
r

Rafa

06/30/2022, 2:05 PM
yeap... but I will do the process again .1 minute
Lucas Rodriguez

Lucas Rodriguez

06/30/2022, 2:10 PM
OK, if it still doesn't work, then you can also try (from the Ubuntu agent):
$ curl --cacert ./fleet_osquery.pem <https://ip:8080/version>
(To check any issues with the generated certificate itself.)
r

Rafa

06/30/2022, 2:22 PM
Noob question:
openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 -nodes \
  -keyout /tmp/server.key -out /tmp/server.cert -subj "/CN=SERVER_NAME" \
  -addext "subjectAltName=DNS:SERVER_NAME"
SERVER_NAME in my case would be the ec2 ip?
Lucas Rodriguez

Lucas Rodriguez

06/30/2022, 2:25 PM
IIRC
subjectAltName=DNS:SERVER_NAME
it should actually be
subjectAltName=IP:$SERVER_IP
r

Rafa

06/30/2022, 2:26 PM
Thanks! And in the CN=SERVER_NAME ?
Lucas Rodriguez

Lucas Rodriguez

06/30/2022, 2:27 PM
Try with the IP too
r

Rafa

06/30/2022, 2:34 PM
openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 -nodes \ -keyout /tmp/server.key -out /tmp/server.cert -subj "/CN=$IP" \ -addext "subjectAltName=IPšŸ˜’IP"
2:34 PM
I create with this command
2:35 PM
Now the accessing the UI is secure
2:36 PM
but the problem is the same: Jun 30 11:34:13 orbit[3659248]: W0630 11:34:13.101204 3659248 tls_enroll.cpp:101] Failed enrollment request to httpsšŸ˜•/:8080/api/v1/osquery/enroll (Request error: certificate verify failed) retrying...
Lucas Rodriguez

Lucas Rodriguez

06/30/2022, 2:37 PM
The curl command works now?
r

Rafa

06/30/2022, 2:44 PM
Yeap! curl --cacert ./fleet_.pem https://ip:8080/version { "version": "4.16.0", "branch": "HEAD", "revision": "865ab32d03c37e8a74e811bc5ac697202f14e455", "go_version": "go1.17.8", "build_date": "2022-06-21", "build_user": "runner" }
Lucas Rodriguez

Lucas Rodriguez

06/30/2022, 2:45 PM
OK, and to double check, you re-generated the package with such cert and re-installed?
2:48 PM
If this is a non-production test, then this can be fixed simply by adding
fleetctl package ... --insecure ...
(which will fix any certificate errors but it's not recommended for production environments.) I can suggest the above and then, once all it's working and tested, configure a proper certificate (not self-signed) for Fleet.
r

Rafa

06/30/2022, 2:50 PM
Yeap, I re-generated the package and use the pem download using UI
2:52 PM
orbit[3665440]: 2022-06-30T11:51:54-03:00 ERR run orbit failed error="write server cert: open /tmp/fleet.crt: permission denied"
2:52 PM
with --insecure
Lucas Rodriguez

Lucas Rodriguez

06/30/2022, 2:52 PM
Do not provide the certificate
--fleet-certificate
when using
--insecure
2:53 PM
Mhm... not sure what's the permission error, let me check
2:54 PM
Isn't Orbit running as root? (it should have access to
/tmp
, right?)
r

Rafa

06/30/2022, 2:56 PM
root 3666950 19.6 0.0 713056 14312 ? Ssl 11:56 0:00 /opt/orbit/bin/orbit/orbit
2:56 PM
yeap, that is the crazy thing
Lucas Rodriguez

Lucas Rodriguez

06/30/2022, 2:59 PM
(The
--insecure
mode creates a certificate in
/tmp/fleet.crt
.)
2:59 PM
Q: Is the goal to test the Fleet deployment before creating a proper certificate for it?
r

Rafa

06/30/2022, 3:01 PM
Yeap!
3:02 PM
I forced the creation /tmp/fleet.crt with privileges and problem continues...
Lucas Rodriguez

Lucas Rodriguez

06/30/2022, 3:03 PM
Orbit needs to create such cert on the fly
r

Rafa

06/30/2022, 3:37 PM
one more thing: when I installed the connector, from which file orbit read the configs?
Lucas Rodriguez

Lucas Rodriguez

06/30/2022, 3:42 PM
Some configs can be set at
fleetctl package
generation time (see
fleetctl package --osquery-flagfile flagfile.txt
option), other options can be set via the Fleet UI (in the Settings -> "Global agent options").
3:46 PM
Let me know if that makes sense
3:47 PM
r

Rafa

06/30/2022, 3:48 PM
Found it
3:49 PM
there is another certificate in the middle
3:49 PM
I am fixing this and let you know
Lucas Rodriguez

Lucas Rodriguez

06/30/2022, 3:50 PM
OK
r

Rafa

06/30/2022, 3:56 PM
worked!
3:56 PM
thanks a lot!
Lucas Rodriguez

Lucas Rodriguez

06/30/2022, 3:56 PM
Cool. Glad to be of help!
r

Rafa

06/30/2022, 4:45 PM
Just one thing more: {"component":"http","err":"error in query ingestion || error in query ingestion || error in query ingestion || error in query ingestion || error in query ingestion || error in query ingestion || error in query ingestion || getting app config: selecting app config: context canceled","ingestion-err":"ingest detail query: selecting app config: context canceled","ip_addr":"IP:13576","level":"error","method":"POST","took":"15.562593584s","ts":"2022-06-30T16:43:53.030188861Z","uri":"/api/v1/osquery/distributed/write","x_for_ip_addr":""}
4:46 PM
any idea what could be the problem?
Lucas Rodriguez

Lucas Rodriguez

06/30/2022, 5:07 PM
context canceled
errors usually are due: ā€¢ Slow database, and/or ā€¢ Configured timeouts in osquery, a load balancer or database (
took
says 15s so a guess is that there's a 15s timeout somewhere)
r

Rafa

06/30/2022, 5:28 PM
strange
5:28 PM
because I put in debug mode
5:28 PM
get the request and send manually
5:28 PM
and it worked
5:28 PM
(fleet ui is showing everything I sent)
5:39 PM
scaling db solved =D
6:10 PM
And last thing: I tried everything: scan host, adding policies, queries, force a rescan and everything works fine. Just one thing is not working: when I run query in the UI it stays forever in loading and in the console this message appears: WebSocket connection to 'wss://ip:8080/api/v1/fleet/results/109/u2d4l2fp/websocket' failed: POST https://ip:8080/api/v1/fleet/results/109/b4sqcld0/xhr_streaming?t=1656612540211 405
6:10 PM
But, from the side of the machine, the request is received and answered correctly.
Lucas Rodriguez

Lucas Rodriguez

06/30/2022, 6:13 PM
It might be an issue with websockets ("live queries" use websocket to connect to Fleet)
6:13 PM
As in something in the infrastructure not allowing websockets traffic.
6:13 PM
6:14 PM
Actually the user in that issue got a 405, same as you. Please take a look at the thread in that issue. (We will be adding an entry for this in the FAQ)
r

Rafa

06/30/2022, 6:22 PM
thanks!