hey gang! Wondering if I could get some support with an issue I'm having with a Google Calendar based update automation. I have my pipeline configured to support this and run a script on a host when the Google Calendar event occurs. The events are all getting created fine but when the meeting starts, Fleet seems to delete the event from the devices status page and the script never gets run. Give it an hour and the a new calendar event is created in Google for the following week. If I move the event up to the current day, the device status page updates with the new time, but when the event starts the event is removed from the Fleet status page, rinse and repeat

@Billy H Thank you for the details. I'm looking into this for you.

Thanks Rebecca!

@Billy H Can you see any errors in your server logs? Also, are you seeing the incoming webhooks trigger in your pipeline?

If you tell me where to look I can find the server logs! This is hosted serverless with Fargate in AWS based on the Fleet deploy code here: https://github.com/fleetdm/fleet/tree/main/terraform I also have an AWS API Gateway to accept the webhooks. The API gateway I have waits for that calendar based automation trigger and sends the script to run on the device but, no, my API gateway never receives any automation trigger from Fleet when the calendar event starts. Plenty of other automation triggers from Fleet hit the API gateway but not the calendar event one

Terraform logs usually go to Cloudwatch.

So I'm checking the logs in CloudWatch and none of them are producing logs when the meeting occurs. The two lamda log groups are for my API gateways and neither of them are receiving any requests during the calendar event

@Billy H This may be a potential bug, considering this integration was working for you prior to your 4.60 update. Can you check the fleetd logs for related errors as well?

I'm checking on the device in the console fleet-desktop.log and not seeing any indication of errors or even any logs from when the event started. But I imagine this is not what you are talking about, are you talking about the logs located in the path listed in the agent options for the variable

logger_path

?

@Billy H Those were your Fleet Desktop logs, but were worth a look anyway. The

logger_path

tells you where your osquery result logs go. What you're looking for are your fleetd agent logs. If you're on a Mac, you can locate and read these files by using these commands:

% sudo cat /private/var/log/orbit/orbit.stderr.log

% sudo cat /private/var/log/orbit/orbit.stdout.log

They have their own

stderr

and

stdout

files.

just had the calendar event disappear and there are 0 new log entries in

stderr

or

stdout

. I think this issue is exclusively on the server side

@Billy H Of course! I will look into the AWS settings you mentioned for enabling debugging.

so I have good news and bad news. The good news is that this seems to have fixed itself! The bad news is I don't know why... After I enabled the debug and json logging, the events started triggering the scripts again... Maybe it finally saw that I was watching and decided it wanted to behave!

@Billy H That's interesting... Glad that it is resolved, but please let us know if it decides to stop behaving and rebels again! calendar fire fine

@Rebecca Cowart I have more logs for you this time! These are the logs I pulled from ECS in AWS and sanitized for privacy sake:

March 06, 2025 at 16:37 (UTC-5:00) {"cron":"calendar","instanceID":"pAwca9XyaedS+FS6QaIzqAyJUggz1BfEy2TaO921R9kkT8KAnmiWmUEAsTRaTdx5dycRh8YgBViYo6pKIbhniQ==","level":"info","schedule":"calendar","status":"completed","ts":"2025-03-06T21:37:09.770231044Z"}
March 06, 2025 at 16:37 (UTC-5:00) {"cron":"calendar","instanceID":"pAwca9XyaedS+FS6QaIzqAyJUggz1BfEy2TaO921R9kkT8KAnmiWmUEAsTRaTdx5dycRh8YgBViYo6pKIbhniQ==","level":"debug","msg":"4m59.22970851s remaining until next tick","schedule":"calendar","ts":"2025-03-06T21:37:09.770331267Z"}
March 06, 2025 at 16:37 (UTC-5:00) {"cron":"calendar","level":"debug","msg":"failing_hosts","team_id":12,"took":"601.643266ms","ts":"2025-03-06T21:37:09.766463288Z"}
March 06, 2025 at 16:37 (UTC-5:00) {"cron":"calendar","host_ids":"[7 60 73]","level":"debug","msg":"no example.com Google account associated with the hosts","team_id":12,"ts":"2025-03-06T21:37:09.766520243Z"}
March 06, 2025 at 16:37 (UTC-5:00) {"cron":"calendar","failing_hosts":1,"failing_hosts_without_associated_email":3,"level":"debug","msg":"summary","passing_hosts":0,"team_id":12,"ts":"2025-03-06T21:37:09.16460976Z"}
March 06, 2025 at 16:37 (UTC-5:00) {"cron":"calendar","level":"debug","msg":"passing_hosts","team_id":12,"took":"104.164µs","ts":"2025-03-06T21:37:09.164755513Z"}
March 06, 2025 at 16:37 (UTC-5:00) {"cron":"calendar","instanceID":"pAwca9XyaedS+FS6QaIzqAyJUggz1BfEy2TaO921R9kkT8KAnmiWmUEAsTRaTdx5dycRh8YgBViYo6pKIbhniQ==","jobID":"calendar_events","level":"debug","msg":"starting","schedule":"calendar","ts":"2025-03-06T21:37:09.154010081Z"}
March 06, 2025 at 16:37 (UTC-5:00) {"cron":"calendar","instanceID":"QWkZehqXw/eV19k1YD3nXZYoqVQ25Id+cLEyw6ZN1UTyDh9xaeLmZqrn1vLC0Q8IkC//Q8prbZEqy6ZndhC/PQ==","level":"debug","msg":"not the lock leader, skipping","schedule":"calendar","ts":"2025-03-06T21:37:09.15003901Z"}
March 06, 2025 at 16:37 (UTC-5:00) {"cron":"calendar","instanceID":"QWkZehqXw/eV19k1YD3nXZYoqVQ25Id+cLEyw6ZN1UTyDh9xaeLmZqrn1vLC0Q8IkC//Q8prbZEqy6ZndhC/PQ==","level":"debug","msg":"unable to acquire lock","schedule":"calendar","ts":"2025-03-06T21:37:09.150093763Z"}
March 06, 2025 at 16:37 (UTC-5:00) {"cron":"calendar","instanceID":"QWkZehqXw/eV19k1YD3nXZYoqVQ25Id+cLEyw6ZN1UTyDh9xaeLmZqrn1vLC0Q8IkC//Q8prbZEqy6ZndhC/PQ==","level":"debug","msg":"4m59.849894166s remaining until next tick","schedule":"calendar","ts":"2025-03-06T21:37:09.150113103Z"}
March 06, 2025 at 16:37 (UTC-5:00) {"cron":"calendar","instanceID":"QWkZehqXw/eV19k1YD3nXZYoqVQ25Id+cLEyw6ZN1UTyDh9xaeLmZqrn1vLC0Q8IkC//Q8prbZEqy6ZndhC/PQ==","level":"debug","msg":"done, tick received","schedule":"calendar","ts":"2025-03-06T21:37:09.140561317Z"}
March 06, 2025 at 16:37 (UTC-5:00) {"cron":"calendar","instanceID":"pAwca9XyaedS+FS6QaIzqAyJUggz1BfEy2TaO921R9kkT8KAnmiWmUEAsTRaTdx5dycRh8YgBViYo6pKIbhniQ==","level":"info","schedule":"calendar","status":"pending","ts":"2025-03-06T21:37:09.137924647Z"}
March 06, 2025 at 16:37 (UTC-5:00) {"cron":"calendar","instanceID":"pAwca9XyaedS+FS6QaIzqAyJUggz1BfEy2TaO921R9kkT8KAnmiWmUEAsTRaTdx5dycRh8YgBViYo6pKIbhniQ==","jobID":"calendar_events_cleanup","level":"debug","msg":"starting","schedule":"calendar","ts":"2025-03-06T21:37:09.13796358Z"}
March 06, 2025 at 16:37 (UTC-5:00) {"cron":"calendar","instanceID":"pAwca9XyaedS+FS6QaIzqAyJUggz1BfEy2TaO921R9kkT8KAnmiWmUEAsTRaTdx5dycRh8YgBViYo6pKIbhniQ==","level":"debug","msg":"done, tick received","schedule":"calendar","ts":"2025-03-06T21:37:09.12706511Z"}
March 06, 2025 at 16:32 (UTC-5:00) {"cron":"calendar","instanceID":"pAwca9XyaedS+FS6QaIzqAyJUggz1BfEy2TaO921R9kkT8KAnmiWmUEAsTRaTdx5dycRh8YgBViYo6pKIbhniQ==","level":"info","schedule":"calendar","status":"completed","ts":"2025-03-06T21:32:09.175765132Z"}
March 06, 2025 at 16:32 (UTC-5:00) {"cron":"calendar","instanceID":"pAwca9XyaedS+FS6QaIzqAyJUggz1BfEy2TaO921R9kkT8KAnmiWmUEAsTRaTdx5dycRh8YgBViYo6pKIbhniQ==","level":"debug","msg":"4m59.824171619s remaining until next tick","schedule":"calendar","ts":"2025-03-06T21:32:09.175865957Z"}
March 06, 2025 at 16:32 (UTC-5:00) {"cron":"calendar","level":"debug","msg":"failing_hosts","team_id":12,"took":"6.20553ms","ts":"2025-03-06T21:32:09.169928036Z"}
March 06, 2025 at 16:32 (UTC-5:00) {"cron":"calendar","host_ids":"[7 60 73]","level":"debug","msg":"no example.com Google account associated with the hosts","team_id":12,"ts":"2025-03-06T21:32:09.170088494Z"}
March 06, 2025 at 16:32 (UTC-5:00) {"cron":"calendar","failing_hosts":1,"failing_hosts_without_associated_email":3,"level":"debug","msg":"summary","passing_hosts":0,"team_id":12,"ts":"2025-03-06T21:32:09.163430537Z"}
March 06, 2025 at 16:32 (UTC-5:00) {"cron":"calendar","level":"debug","msg":"passing_hosts","team_id":12,"took":"43.37µs","ts":"2025-03-06T21:32:09.163645227Z"}
March 06, 2025 at 16:32 (UTC-5:00) {"cron":"calendar","instanceID":"pAwca9XyaedS+FS6QaIzqAyJUggz1BfEy2TaO921R9kkT8KAnmiWmUEAsTRaTdx5dycRh8YgBViYo6pKIbhniQ==","jobID":"calendar_events","level":"debug","msg":"starting","schedule":"calendar","ts":"2025-03-06T21:32:09.151412409Z"}
March 06, 2025 at 16:32 (UTC-5:00) {"cron":"calendar","instanceID":"QWkZehqXw/eV19k1YD3nXZYoqVQ25Id+cLEyw6ZN1UTyDh9xaeLmZqrn1vLC0Q8IkC//Q8prbZEqy6ZndhC/PQ==","level":"debug","msg":"not the lock leader, skipping","schedule":"calendar","ts":"2025-03-06T21:32:09.140353604Z"}
March 06, 2025 at 16:32 (UTC-5:00) {"cron":"calendar","instanceID":"QWkZehqXw/eV19k1YD3nXZYoqVQ25Id+cLEyw6ZN1UTyDh9xaeLmZqrn1vLC0Q8IkC//Q8prbZEqy6ZndhC/PQ==","level":"debug","msg":"unable to acquire lock","schedule":"calendar","ts":"2025-03-06T21:32:09.140399065Z"}
March 06, 2025 at 16:32 (UTC-5:00) {"cron":"calendar","instanceID":"QWkZehqXw/eV19k1YD3nXZYoqVQ25Id+cLEyw6ZN1UTyDh9xaeLmZqrn1vLC0Q8IkC//Q8prbZEqy6ZndhC/PQ==","level":"debug","msg":"4m59.859590633s remaining until next tick","schedule":"calendar","ts":"2025-03-06T21:32:09.140417824Z"}
March 06, 2025 at 16:32 (UTC-5:00) {"cron":"calendar","instanceID":"pAwca9XyaedS+FS6QaIzqAyJUggz1BfEy2TaO921R9kkT8KAnmiWmUEAsTRaTdx5dycRh8YgBViYo6pKIbhniQ==","level":"info","schedule":"calendar","status":"pending","ts":"2025-03-06T21:32:09.130920203Z"}
March 06, 2025 at 16:32 (UTC-5:00) {"cron":"calendar","instanceID":"pAwca9XyaedS+FS6QaIzqAyJUggz1BfEy2TaO921R9kkT8KAnmiWmUEAsTRaTdx5dycRh8YgBViYo6pKIbhniQ==","jobID":"calendar_events_cleanup","level":"debug","msg":"starting","schedule":"calendar","ts":"2025-03-06T21:32:09.130964056Z"}
March 06, 2025 at 16:32 (UTC-5:00) {"cron":"calendar","instanceID":"QWkZehqXw/eV19k1YD3nXZYoqVQ25Id+cLEyw6ZN1UTyDh9xaeLmZqrn1vLC0Q8IkC//Q8prbZEqy6ZndhC/PQ==","level":"debug","msg":"done, tick received","schedule":"calendar","ts":"2025-03-06T21:32:09.127605434Z"}
March 06, 2025 at 16:32 (UTC-5:00) {"cron":"calendar","instanceID":"pAwca9XyaedS+FS6QaIzqAyJUggz1BfEy2TaO921R9kkT8KAnmiWmUEAsTRaTdx5dycRh8YgBViYo6pKIbhniQ==","level":"debug","msg":"done, tick received","schedule":"calendar","ts":"2025-03-06T21:32:09.118714779Z"}

Yesterday when I tested, the script never ran. Today when I test the script ran from the calendar event but only 7 minutes after the event started. Usually it starts within the first 30 seconds. Seeing some odd errors in the logs here, namely:

no <http://example.com|example.com> Google account associated with the hosts

@Billy H (osquery) Thank you for sharing these! Are there any hosts, among the hosts that you scheduled the event for, that don't have gmail addresses listed in their Used By field?

Nope! They all have to have an email address associated with them for the calendar event to even show up. They wont get the event until an account is associated with the computer

I think this is based on the Google profile, not the local macOS account?

That's what I figured but I wanted to put it out there just in case

@Billy H (osquery) Can you check on the Hosts with ids of 7, 60, and 73 and make sure they have a Google account listed in their Used By field?

Hey! Sorry busy week last week. Yes I was able to confirm that there was a google account listed in their Used By field

@Billy H These google accounts all have the same matching domain that you declared as your primary domain while following the instructions, correct?

that is correct! they all have the matching domain

@Billy H Focusing on the webhook that isn't firing at all: The webhook will only fire upon a newly failing host for that policy. Is there a chance that everyone was already failing that policy before you added the GC integration to it?

Ah that might be doing it!

@Billy H A good way to test would be to create a policy that is purely failing (like

SELECT 1 WHERE 1=2

) with the immediate implementation of the Google Calendar integration, before any hosts have a chance to fail it. Adding on an integration does not reset the host's pass/fail state for that policy.