Hi I m testing FleetDM premium I m connected macOS with flee osquery #fleet

Hi! I'm testing FleetDM premium. I'm connected mac...

Secf

10/18/2024, 12:16 PM

Hi! I'm testing FleetDM premium. I'm connected macOS with fleet, uploaded test script and from GUI can't run script: "Scripts are disabled for this host. To run scripts, deploy the fleetd agent with scripts enabled." Maybe exists parameter I need to turn on?

Jacob Burley

10/18/2024, 12:29 PM

Did you build your macOS installer with scripting enabled?

Secf

10/18/2024, 12:34 PM

@Jacob Burley yes, using this command: fleetctl package --type=pkg --enable-scripts --fleet-desktop --fleet-url=https://... --enroll-secret=...

Secf

10/18/2024, 12:34 PM

same issue also on windows hosts

Zay Hanlon

10/18/2024, 2:27 PM

@Secf what version of Fleet are you running?

Secf

10/18/2024, 3:01 PM

@Zay Hanlon Fleet 4.58.0

Zay Hanlon

10/18/2024, 3:33 PM

Can you check if someone marked off 'Disable scripts' in org settings?

Zay Hanlon

10/18/2024, 3:33 PM

Screenshot 2024-10-18 at 11.33.11 AM.png

Secf

10/18/2024, 4:05 PM

@Zay Hanlon same, this option also marked off

Rebecca Cowart

10/18/2024, 5:44 PM

@Secf Can you try running this query?

Copy code

SELECT * FROM orbit_info

If it returns that the scripts are enabled, you could try refetching the hosts.

Secf

10/19/2024, 4:10 PM

@Rebecca Cowart After run the query: "SELECT * FROM orbit_info" in last_recorded_error table i noticed these messages: 2024-10-19T135913Z: Post "https://../api/fleet/orbit/device_token": dial tcp: lookup ...: no such host 2024-10-19T081735Z: Post "https://../api/fleet/orbit/device_token": dial tcp ..443 connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host...o respond. (truncated) and under scripts_enabled - int 1 (shown on screenshot)

Rebecca Cowart

10/21/2024, 5:48 PM

I will look into those errors further for you. In the meantime, since your response came back positive for scripts_enabled, could you try refetching your hosts to see if that will remedy the initial issue?

Secf

10/23/2024, 11:25 AM

@Rebecca Cowart I just did a refetch and re-ran the script, but still the error. Even deleted the host and re-entered fleetctl... --enable-scripts...

Secf

10/23/2024, 2:40 PM

@Rebecca Cowart Can you help please?

Secf

10/25/2024, 6:34 AM

Please tell me if there is any information for me?

Rebecca Cowart

10/25/2024, 12:01 PM

@Secf I'm still looking into this issue for you and will get back to you today. 🙂 Sorry for the delay. Our support team has been at a company offsite this week.

Secf

10/25/2024, 12:03 PM

@Rebecca Cowart thanks, will be waiting

Rebecca Cowart

10/25/2024, 12:35 PM

@Secf Can you list out the infrastructure that you're using to run your Fleet instance and their versions?

Secf

10/25/2024, 12:56 PM

@Rebecca Cowart Fleet 4.58.0, go1.23.1, redis server v6.0.16, MySQL v8.0.39, deployed on Ubuntu 22.04.5 (LTS)

Secf

10/25/2024, 12:59 PM

Osquery: 5.13.1, orbit: 1.34.0

Rebecca Cowart

10/25/2024, 1:17 PM

@Secf I have escalated the issue and will return shortly. Thank you for the information!

Kathy Satterlee

10/25/2024, 2:45 PM

@Secf Can you share the fleetd logs for the host in question? https://fleetdm.com/guides/enroll-hosts#finding-fleetd-logs

Secf

10/25/2024, 4:03 PM

@Kathy Satterlee These logs from macOS (/private/var/log/orbit/orbit.stderr.log): P.S: /private/var/log/orbit/orbit.stdout.log - is empty

Copy code

2024-10-24T11:20:33+03:00 INF token TTL expired, rotating token
2024-10-24T11:21:08+03:00 ERR error rotating token error="saving token after 3 attempts: POST /api/fleet/orbit/device_token: Post \"<https://fleet-test.com/api/fleet/orbit/device_token\|https://fleet-test.com/api/fleet/orbit/device_token\>": dial tcp x.x.x.x:443: connect: operation timed out"
2024-10-24T11:21:08+03:00 INF token TTL expired, rotating token
2024-10-24T11:21:40+03:00 INF network error error="POST /api/fleet/orbit/config: Post \"<https://fleet-test.com/api/fleet/orbit/config\|https://fleet-test.com/api/fleet/orbit/config\>": dial tcp x.x.x.x:443: connect: operation timed out"
2024-10-24T11:21:43+03:00 ERR error rotating token error="saving token after 3 attempts: POST /api/fleet/orbit/device_token: Post \"<https://fleet-test.com/api/fleet/orbit/device_token\|https://fleet-test.com/api/fleet/orbit/device_token\>": dial tcp x.x.x.x:443: connect: operation timed out"
2024-10-24T11:21:43+03:00 INF token TTL expired, rotating token
2024-10-24T11:22:19+03:00 ERR error rotating token error="saving token after 3 attempts: POST /api/fleet/orbit/device_token: Post \"<https://fleet-test.com/api/fleet/orbit/device_token\|https://fleet-test.com/api/fleet/orbit/device_token\>": dial tcp x.x.x.x:443: connect: operation timed out"
2024-10-24T11:22:19+03:00 INF token TTL expired, rotating token
2024-10-24T11:22:54+03:00 ERR error rotating token error="saving token after 3 attempts: POST /api/fleet/orbit/device_token: Post \"<https://fleet-test.com/api/fleet/orbit/device_token\|https://fleet-test.com/api/fleet/orbit/device_token\>": dial tcp x.x.x.x:443: connect: operation timed out"
2024-10-24T11:22:54+03:00 INF token TTL expired, rotating token
2024-10-24T12:22:58+03:00 INF token TTL expired, rotating token
2024-10-24T13:22:58+03:00 INF token TTL expired, rotating token
2024-10-24T14:31:44+03:00 INF token TTL expired, rotating token
2024-10-24T15:33:19+03:00 INF token TTL expired, rotating token
2024-10-24T16:33:19+03:00 INF token TTL expired, rotating token
2024-10-24T18:06:13+03:00 INF token TTL expired, rotating token
2024-10-24T19:37:05+03:00 INF token TTL expired, rotating token
2024-10-24T20:47:55+03:00 INF token TTL expired, rotating token
2024-10-24T21:47:55+03:00 INF token TTL expired, rotating token
2024-10-24T22:47:55+03:00 INF token TTL expired, rotating token
2024-10-24T23:48:09+03:00 INF token TTL expired, rotating token
2024-10-25T01:27:59+03:00 INF token TTL expired, rotating token
2024-10-25T02:38:09+03:00 INF token TTL expired, rotating token
2024-10-25T04:01:55+03:00 INF token TTL expired, rotating token
2024-10-25T05:18:02+03:00 INF token TTL expired, rotating token
2024-10-25T07:21:22+03:00 INF network error error="POST /api/fleet/orbit/config: Post \"<https://fleet-test.com/api/fleet/orbit/config\|https://fleet-test.com/api/fleet/orbit/config\>": read tcp 192.168.1.10:57324->x.x.x.x:443: read: operation timed out"
2024-10-25T07:41:49+03:00 INF token TTL expired, rotating token
2024-10-25T08:42:52+03:00 INF token TTL expired, rotating token
2024-10-25T09:42:52+03:00 INF token TTL expired, rotating token
2024-10-25T10:10:12+03:00 INF periodic check of token failed, initiating rotation error="HEAD /api/latest/fleet/device/2404791b-23b7-4543-8e99-72401f90ec6f/ping: Head \"<https://fleet-test.com/api/latest/fleet/device/2404791b-23b7-4543-8e99-72401f90ec6f/ping\|https://fleet-test.com/api/latest/fleet/device/2404791b-23b7-4543-8e99-72401f90ec6f/ping\>": net/http: TLS handshake timeout"
2024-10-25T12:58:04+03:00 INF network error error="POST /api/fleet/orbit/config: Post \"<https://fleet-test.com/api/fleet/orbit/config\|https://fleet-test.com/api/fleet/orbit/config\>": read tcp 192.168.1.10:57781->x.x.x.x:443: read: operation timed out"
2024-10-25T13:59:30+03:00 INF token TTL expired, rotating token
2024-10-25T14:59:30+03:00 INF token TTL expired, rotating token
2024-10-25T15:59:30+03:00 INF token TTL expired, rotating token
2024-10-25T17:03:01+03:00 INF token TTL expired, rotating token
2024-10-25T18:40:31+03:00 INF token TTL expired, rotating token

Secf

10/25/2024, 4:05 PM

And this url https://fleet-test.com/api/fleet/orbit/config\ responded 405

Tim Lee

10/25/2024, 5:20 PM

possible dns issue? does

nc -z <http://fleet-test.com|fleet-test.com> 443

work from the macOS host?

Secf

10/25/2024, 5:36 PM

@Tim Lee No, DNS is working, after execution above command -> Connection to fleet-test.com port 443 [tcp/https] succeeded!

Rebecca Cowart

10/25/2024, 5:39 PM

@Secf If possible, can you share your Fleet server logs?

Secf

10/25/2024, 5:43 PM

@Rebecca Cowart When i execute command: systemctl status orbit on Ubuntu server (where Fleet deployed) it answering -> Unit orbit.service could not be found.

Secf

10/25/2024, 5:44 PM

Maybe problem here, with orbit?

Tim Lee

10/25/2024, 5:49 PM

it seems

orbit

and

fleet-desktop

are both unable to connect to the server. Possibly more info in the server logs. the orbit service only exists on devices, the server service i believe is named

fleet

Tim Lee

10/25/2024, 5:49 PM

for reference, is this deployed on a local VM?

Secf

10/25/2024, 5:50 PM

@Tim Lee no, it's deployed on ec2 instance of AWS (with Ubuntu OS)

Secf

10/25/2024, 5:51 PM

and it's my /etc/fleet/fleet.conf file:

Copy code

mysql:
  address: localhost:3306
  database: fleet
  username: fleet
  password: 
  max_open_conns: 1000
  mysql_max_idle_conns: 1000
redis:
  address: 127.0.0.1:6379
  redis_username: 
  redis_password: 
server:
  address: 127.0.0.1:8080
  # this is certbot certs
  # not automated copy them from certbot
  # need to do it by hands
  cert: /etc/fleet/certs/server.crt
  key: /etc/fleet/certs/server.key
  private_key: 
  websockets_allow_unsafe_origin: true
logging:
  json: true
osquery:
  label_query_update_interval: 12h
  detail_update_interval: 15m
  host_identifier: instance
filesystem:
  status_log_file: /var/log/osquery/status.log
  result_log_file: /var/log/osquery/result.log
  enable_log_rotation: true
license:
  key: 
mdm:
  windows_wstep_identity_cert: /etc/fleet/certs/fleet-mdm-win-wstep.crt
  windows_wstep_identity_key: /etc/fleet/certs/fleet-mdm-win-wstep.key

Tim Lee

10/25/2024, 5:55 PM

here's a reference: https://fleetdm.com/docs/deploy/reference-architectures#run-with-systemd you can get server logs from

Copy code

sudo journalctl -u fleet.service -f

or possibly cloudwatch if deployed on EC2

Secf

10/25/2024, 6:01 PM

Copy code

Oct 25 17:56:04 fleet[1599]: {"cron":"integrations","level":"info","msg":"no cooldowns to process","ts":"2024-10-25T17:56:04.44230538Z"}
Oct 25 17:56:04 fleet[1599]: {"cron":"integrations","instanceID":"YKRrdQQBOmCPHG2cFypm382a83FsQjAd6YbC8uJiH9UxkDXTjD9jSFHKa4RCEgu7vNBvZbDzTgrRAf4kOxSt/A==","level":"info","schedule":"integrations","status":"completed","ts":"2024-10-25T17:56:04.446250684Z"}
Oct 25 17:56:04 fleet[1599]: {"cron":"apple_mdm_apns_pusher","instanceID":"YKRrdQQBOmCPHG2cFypm382a83FsQjAd6YbC8uJiH9UxkDXTjD9jSFHKa4RCEgu7vNBvZbDzTgrRAf4kOxSt/A==","level":"info","schedule":"apple_mdm_apns_pusher","status":"pending","ts":"2024-10-25T17:56:04.472148254Z"}
Oct 25 17:56:04 fleet[1599]: {"cron":"apple_mdm_apns_pusher","instanceID":"YKRrdQQBOmCPHG2cFypm382a83FsQjAd6YbC8uJiH9UxkDXTjD9jSFHKa4RCEgu7vNBvZbDzTgrRAf4kOxSt/A==","level":"info","schedule":"apple_mdm_apns_pusher","status":"completed","ts":"2024-10-25T17:56:04.476894292Z"}
Oct 25 17:56:04 fleet[1599]: {"component":"nanodep-syncer","cron":"apple_mdm_dep_profile_assigner","instanceID":"YKRrdQQBOmCPHG2cFypm382a83FsQjAd6YbC8uJiH9UxkDXTjD9jSFHKa4RCEgu7vNBvZbDzTgrRAf4kOxSt/A==","level":"info","schedule":"apple_mdm_dep_profile_assigner","status":"pending","ts":"2024-10-25T17:56:04.533700345Z"}
Oct 25 17:56:04 fleet[1599]: {"component":"nanodep-syncer","cron":"apple_mdm_dep_profile_assigner","instanceID":"YKRrdQQBOmCPHG2cFypm382a83FsQjAd6YbC8uJiH9UxkDXTjD9jSFHKa4RCEgu7vNBvZbDzTgrRAf4kOxSt/A==","level":"info","schedule":"apple_mdm_dep_profile_assigner","status":"completed","ts":"2024-10-25T17:56:04.537580555Z"}
Oct 25 17:56:13 fleet[1599]: {"cron":"mdm_apple_profile_manager","instanceID":"YKRrdQQBOmCPHG2cFypm382a83FsQjAd6YbC8uJiH9UxkDXTjD9jSFHKa4RCEgu7vNBvZbDzTgrRAf4kOxSt/A==","level":"info","schedule":"mdm_apple_profile_manager","status":"pending","ts":"2024-10-25T17:56:13.539888484Z"}
Oct 25 17:56:13 fleet[1599]: {"cron":"mdm_apple_profile_manager","instanceID":"YKRrdQQBOmCPHG2cFypm382a83FsQjAd6YbC8uJiH9UxkDXTjD9jSFHKa4RCEgu7vNBvZbDzTgrRAf4kOxSt/A==","level":"info","schedule":"mdm_apple_profile_manager","status":"completed","ts":"2024-10-25T17:56:13.546960204Z"}
Oct 25 17:56:23 fleet[1599]: {"component":"iphone-ipad-refetcher","cron":"apple_mdm_iphone_ipad_refetcher","instanceID":"YKRrdQQBOmCPHG2cFypm382a83FsQjAd6YbC8uJiH9UxkDXTjD9jSFHKa4RCEgu7vNBvZbDzTgrRAf4kOxSt/A==","level":"info","schedule":"apple_mdm_iphone_ipad_refetcher","status":"pending","ts":"2024-10-25T17:56:23.46988026Z"}
Oct 25 17:56:23 fleet[1599]: {"component":"iphone-ipad-refetcher","cron":"apple_mdm_iphone_ipad_refetcher","instanceID":"YKRrdQQBOmCPHG2cFypm382a83FsQjAd6YbC8uJiH9UxkDXTjD9jSFHKa4RCEgu7vNBvZbDzTgrRAf4kOxSt/A==","level":"info","schedule":"apple_mdm_iphone_ipad_refetcher","status":"completed","ts":"2024-10-25T17:56:23.476075717Z"}
Oct 25 17:56:33 fleet[1599]: {"cron":"calendar","instanceID":"YKRrdQQBOmCPHG2cFypm382a83FsQjAd6YbC8uJiH9UxkDXTjD9jSFHKa4RCEgu7vNBvZbDzTgrRAf4kOxSt/A==","level":"info","schedule":"calendar","status":"pending","ts":"2024-10-25T17:56:33.924923873Z"}
Oct 25 17:56:33 fleet[1599]: {"cron":"calendar","instanceID":"YKRrdQQBOmCPHG2cFypm382a83FsQjAd6YbC8uJiH9UxkDXTjD9jSFHKa4RCEgu7vNBvZbDzTgrRAf4kOxSt/A==","level":"info","schedule":"calendar","status":"completed","ts":"2024-10-25T17:56:33.932109788Z"}
Oct 25 17:56:43 fleet[1599]: {"cron":"mdm_apple_profile_manager","instanceID":"YKRrdQQBOmCPHG2cFypm382a83FsQjAd6YbC8uJiH9UxkDXTjD9jSFHKa4RCEgu7vNBvZbDzTgrRAf4kOxSt/A==","level":"info","schedule":"mdm_apple_profile_manager","status":"pending","ts":"2024-10-25T17:56:43.559908038Z"}
Oct 25 17:56:43 fleet[1599]: {"cron":"mdm_apple_profile_manager","instanceID":"YKRrdQQBOmCPHG2cFypm382a83FsQjAd6YbC8uJiH9UxkDXTjD9jSFHKa4RCEgu7vNBvZbDzTgrRAf4kOxSt/A==","level":"info","schedule":"mdm_apple_profile_manager","status":"completed","ts":"2024-10-25T17:56:43.569016791Z"}
Oct 25 17:57:04 fleet[1599]:

Copy code

{"cron":"integrations","instanceID":"YKRrdQQBOmCPHG2cFypm382a83FsQjAd6YbC8uJiH9UxkDXTjD9jSFHKa4RCEgu7vNBvZbDzTgrRAf4kOxSt/A==","level":"info","schedule":"integrations","status":"pending","ts":"2024-10-25T17:57:04.454662097Z"}
Oct 25 17:57:04 fleet[1599]: {"cron":"integrations","level":"info","msg":"no cooldowns to process","ts":"2024-10-25T17:57:04.456109445Z"}

Tim Lee

10/25/2024, 6:03 PM

logs can be pretty verbose, can you send a larger sample size, maybe a text file

Secf

10/25/2024, 6:27 PM

@Tim Lee

fleet_logs.txt

Secf

10/25/2024, 6:28 PM

And also why in GUI of Fleet it shows 2 macOS hosts, though on Fleet only one host (but in "Hosts" shows online and ofline status of one host ), is it normal behavior?)

Tim Lee

10/25/2024, 6:45 PM

Can you explain a bit more about the DNS setup, as

<http://fleet-test.com|fleet-test.com>

is not a publicly accessible domain

Secf

10/25/2024, 6:49 PM

@Tim Lee this is internal DNS, behind corporate VPN

Tim Lee

10/25/2024, 8:29 PM

Interesting, is it a vpn client on the host? Or maybe a vpn client on a hyper visor? Orbit may not be respecting the vpn dns config.

Secf

10/28/2024, 6:46 AM

@Tim Lee On the host, I'm looking on VPN dns config later Also noticed when i'm trying to run Scripts in Fleet GUI - getting 422 HTTP response from server. is this script format correct for macOS in Fleet context?

#!/bin/zsh

echo "test" > ~/test.txt

Secf

10/28/2024, 6:47 AM

I want to test and deploy locally without any VPNs and DNS. Maybe exist fast installation script for it?

Secf

10/29/2024, 7:35 AM

Secf

10/30/2024, 5:53 PM

@Tim Lee any information?

Tim Lee

10/30/2024, 6:09 PM

My guess here is that orbit isn't respecting the VPN client DNS. Maybe updating

/etc/hosts

could be a workaround. Orbit is responsible for script execution, so no surprise on the errors if it cannot connect to Fleet.

Secf

11/01/2024, 8:02 AM

@Tim Lee it's working! Just redeployed via docker compose

5 Views

Open in Slack

Previous Next