re: fleet policy automations, does it ping the webhook each time it checks and a host fails a policy, or does it only do it once per host per policy? e.g. if someone is failing a policy for a week straight, how many times will the webhook get pinged for that specific device?

I believe you'd only get notified on the rising/falling edges. The change in state from failing to passing or passing to failing

That’s not quite right.

if someone is failing a policy for a week straight, how many times will the webhook get pinged for that specific device?

Only once. Fleet checks policies every 24hr (configurable). If there are any “newly failing” hosts, Fleet pings a webhook per policy. “Newly failing” here means if a host has updated from no response to failing or passing to failing since the last check.