https://github.com/osquery/osquery logo
#fleet
Title
# fleet
j

Jason Roberts

02/15/2023, 6:10 PM
Hello, I'm wondering if anyone knows were to begin to troubleshoot this issue. The dashboard is showing operating systems last updated 2 months ago. Not sure why this is no longer updating. Is there some kind of remote website that it has to reach to get this data?
k

Kathy Satterlee

02/15/2023, 6:51 PM
Hi @Jason Roberts. That time reflects the last time the host counts for operating systems were updated. I'd expect that to be updated fairly frequently. Are you seeing any errors in the Fleet logs, specifically related to
cron
jobs?
j

Jason Roberts

02/15/2023, 10:24 PM
i just realized my logging is not writing to the correct place let me fix that will get back to you.
i searched for the word chron but found no errors. Unless it does not have the word cron in the error.
although this is only one node in the autoscaling group its at least been running for a day.
k

Kathy Satterlee

02/16/2023, 6:46 PM
What version of Fleet are you running?
j

Jason Roberts

02/16/2023, 6:47 PM
Copy code
{
  "version": "4.27.0",
  "branch": "HEAD",
  "revision": "b3daf3d7155830d74e18110c319d2b28d2feea71",
  "go_version": "go1.19.4",
  "build_date": "2023-02-04",
  "build_user": "runner"
}
k

Kathy Satterlee

02/16/2023, 7:03 PM
Can you try manually triggering the cleanup job with
fleetctl trigger --name cleanups_then_aggregation
I'd like to see if you get any errors there
j

Jason Roberts

02/16/2023, 7:23 PM
Copy code
./fleetctl trigger --name cleanups_then_aggregation
[!] Conflicts with current status of cleanups_then_aggregation schedule: scheduled run started 1745h59m48.624s ago
k

Kathy Satterlee

02/16/2023, 9:31 PM
That's the culprit! likely for both issues. We're working on a fix to the underlying problem, but here's how you can clean it up now: 1. Clear out the stalled cleanup job by running this query against the fleet database:
Copy code
UPDATE cron_stats SET status = 'expired' WHERE name = 'cleanups_then_aggregation' AND stats_type = 'scheduled' AND status = 'pending'
2. Trigger a new cleanup job:
Copy code
fleetctl trigger --name cleanups_then_aggregation
That should clear everything up and get things moving again.
j

Jason Roberts

02/16/2023, 9:32 PM
against the SQL database?
k

Kathy Satterlee

02/16/2023, 9:33 PM
Yes, sorry. Could have been more clear there.
j

Jason Roberts

02/16/2023, 9:33 PM
lol i figured it looked like sql
Copy code
fleetctl_v4.27.0_macos % ./fleetctl trigger --name cleanups_then_aggregation                                                                   
[+] Sent request to trigger cleanups_then_aggregation schedule
it worked
oh wow that unstuck the host cleanup job too and deleted a bunch of hosts finally
k

Kathy Satterlee

02/17/2023, 12:30 AM
That's awesome!!