Channels
android_tests
apple-silicon
arm-architecture
auditing-warroom
aws
carving
code-review
community-feeds
core
darkbytes
doorman
ebpf
eclecticiq-polylogyx-extension
extensions
file-carving
fim
fleet
fleet-dev
fleetosquery
foundation
fuzzing
general
golang
goquery
infrastructure
jobs
kolide
linen-dev
linux
macos
officehours
osctrl
plugins
process-auditing
querycon
queryhub
random
selfgroup
sql
tls
uptycs
vendor-feeds
website
windows
zeek
zentral
zercurity
Title
j
Jason Roberts
02/15/2023, 6:10 PMHello,
I'm wondering if anyone knows were to begin to troubleshoot this issue. The dashboard is showing operating systems last updated 2 months ago. Not sure why this is no longer updating. Is there some kind of remote website that it has to reach to get this data?
k
Kathy Satterlee
02/15/2023, 6:51 PMHi @Jason Roberts. That time reflects the last time the host counts for operating systems were updated. I'd expect that to be updated fairly frequently. Are you seeing any errors in the Fleet logs, specifically related to
cronj
Jason Roberts
02/15/2023, 10:24 PMi just realized my logging is not writing to the correct place let me fix that will get back to you.
i searched for the word chron but found no errors. Unless it does not have the word cron in the error.
although this is only one node in the autoscaling group its at least been running for a day.
k
Kathy Satterlee
02/16/2023, 6:46 PMWhat version of Fleet are you running?
j
Jason Roberts
02/16/2023, 6:47 PM{
"version": "4.27.0",
"branch": "HEAD",
"revision": "b3daf3d7155830d74e18110c319d2b28d2feea71",
"go_version": "go1.19.4",
"build_date": "2023-02-04",
"build_user": "runner"
}k
Kathy Satterlee
02/16/2023, 7:03 PMCan you try manually triggering the cleanup job with
fleetctl trigger --name cleanups_then_aggregationI'd like to see if you get any errors there
j
Jason Roberts
02/16/2023, 7:23 PM./fleetctl trigger --name cleanups_then_aggregation
[!] Conflicts with current status of cleanups_then_aggregation schedule: scheduled run started 1745h59m48.624s agok
Kathy Satterlee
02/16/2023, 9:31 PMThat's the culprit! likely for both issues. We're working on a fix to the underlying problem, but here's how you can clean it up now:
1. Clear out the stalled cleanup job by running this query against the fleet database:
UPDATE cron_stats SET status = 'expired' WHERE name = 'cleanups_then_aggregation' AND stats_type = 'scheduled' AND status = 'pending'fleetctl trigger --name cleanups_then_aggregationj
Jason Roberts
02/16/2023, 9:32 PMagainst the SQL database?
k
Kathy Satterlee
02/16/2023, 9:33 PMYes, sorry. Could have been more clear there.
j
Jason Roberts
02/16/2023, 9:33 PMlol i figured it looked like sql
fleetctl_v4.27.0_macos % ./fleetctl trigger --name cleanups_then_aggregation
[+] Sent request to trigger cleanups_then_aggregation scheduleit worked
oh wow that unstuck the host cleanup job too and deleted a bunch of hosts finally
k
Kathy Satterlee
02/17/2023, 12:30 AMThat's awesome!!