Channels
#fleet
Title
# fleet
j
Jason Roberts
02/15/2023, 6:10 PMHello,
I'm wondering if anyone knows were to begin to troubleshoot this issue. The dashboard is showing operating systems last updated 2 months ago. Not sure why this is no longer updating. Is there some kind of remote website that it has to reach to get this data?
k
Kathy Satterlee
02/15/2023, 6:51 PMHi @Jason Roberts. That time reflects the last time the host counts for operating systems were updated. I'd expect that to be updated fairly frequently. Are you seeing any errors in the Fleet logs, specifically related to
cronj
Jason Roberts
02/15/2023, 10:24 PMi just realized my logging is not writing to the correct place let me fix that will get back to you.
i searched for the word chron but found no errors. Unless it does not have the word cron in the error.
although this is only one node in the autoscaling group its at least been running for a day.
k
Kathy Satterlee
02/16/2023, 6:46 PMWhat version of Fleet are you running?
j
Jason Roberts
02/16/2023, 6:47 PM Copy code
{
"version": "4.27.0",
"branch": "HEAD",
"revision": "b3daf3d7155830d74e18110c319d2b28d2feea71",
"go_version": "go1.19.4",
"build_date": "2023-02-04",
"build_user": "runner"
}k
Kathy Satterlee
02/16/2023, 7:03 PMCan you try manually triggering the cleanup job with
fleetctl trigger --name cleanups_then_aggregationI'd like to see if you get any errors there
j
Jason Roberts
02/16/2023, 7:23 PM Copy code
./fleetctl trigger --name cleanups_then_aggregation
[!] Conflicts with current status of cleanups_then_aggregation schedule: scheduled run started 1745h59m48.624s agok
Kathy Satterlee
02/16/2023, 9:31 PMThat's the culprit! likely for both issues. We're working on a fix to the underlying problem, but here's how you can clean it up now:
1. Clear out the stalled cleanup job by running this query against the fleet database:
Copy code
UPDATE cron_stats SET status = 'expired' WHERE name = 'cleanups_then_aggregation' AND stats_type = 'scheduled' AND status = 'pending' Copy code
fleetctl trigger --name cleanups_then_aggregationj
Jason Roberts
02/16/2023, 9:32 PMagainst the SQL database?
k
Kathy Satterlee
02/16/2023, 9:33 PMYes, sorry. Could have been more clear there.
j
Jason Roberts
02/16/2023, 9:33 PMlol i figured it looked like sql
Copy code
fleetctl_v4.27.0_macos % ./fleetctl trigger --name cleanups_then_aggregation
[+] Sent request to trigger cleanups_then_aggregation scheduleit worked
oh wow that unstuck the host cleanup job too and deleted a bunch of hosts finally
k
Kathy Satterlee
02/17/2023, 12:30 AMThat's awesome!!