When I have a scheduled query that fails because o...
# general
When I have a scheduled query that fails because of the watchdog process interrupting it for exceeding time or memory, where can I conveniently locate which query has failed? We've got numerous packs of scheduled queries, and it seems tricky to tease out which one's running afoul of the resource constraints.
In Fleet we use the info in the
table. I always recommend folks schedule a query to that table.
@zwass as in having a periodic (hourly?) query there to see the current state of it at snapshots in time?
Yes, just like that!
There's information about what is scheduled, the resources it's been using, watchdog settings, etc.
Hmm, let me give that some thought and look at the schema. Thanks!
Certain classes of out-and-out errors log which query it was, but the scheduled queries interrupted by watchdog do not, which is a little frustrating.
@zwass, also odd regarding that osquery_schedule table is that our details are mostly zeroed out. I'll look up whether that's flag-controlled:
Copy code
name = pack_inci[...]
         query = select * [...]
      interval = 3600
    executions = 0
 last_executed = 0
    denylisted = 0
   output_size =
     wall_time = 0
     user_time = 0
   system_time = 0
average_memory = 0
Hmmm, the info in the
table is since the last startup. Persisting this across restarts is something @sharvil and I have been discussing putting on Fleet's osquery contribution roadmap.
Ah, interesting, that's useful information, since we'd just rolled out updates today, and think osquery restarted most everywhere.
Was poking around the source code, and found that I can do 
flag and get the system to output which query it's executing to the logs.
IIRC you could also turn on verbose logging
ty 1
Yeah, I could, but across a fleet that'd be too much info. This'll get me what I need, the schedule_lognames thing above.