Title
#general
h

HarlanF

01/25/2022, 12:42 AM
When I have a scheduled query that fails because of the watchdog process interrupting it for exceeding time or memory, where can I conveniently locate which query has failed? We've got numerous packs of scheduled queries, and it seems tricky to tease out which one's running afoul of the resource constraints.
zwass

zwass

01/25/2022, 12:46 AM
In Fleet we use the info in the
osquery_schedule
table. I always recommend folks schedule a query to that table.
h

HarlanF

01/25/2022, 12:54 AM
@zwass as in having a periodic (hourly?) query there to see the current state of it at snapshots in time?
zwass

zwass

01/25/2022, 12:54 AM
Yes, just like that!
12:54 AM
There's information about what is scheduled, the resources it's been using, watchdog settings, etc.
h

HarlanF

01/25/2022, 12:54 AM
Hmm, let me give that some thought and look at the schema. Thanks!
12:56 AM
Certain classes of out-and-out errors log which query it was, but the scheduled queries interrupted by watchdog do not, which is a little frustrating.
12:58 AM
@zwass, also odd regarding that osquery_schedule table is that our details are mostly zeroed out. I'll look up whether that's flag-controlled:
name = pack_inci[...]
         query = select * [...]
      interval = 3600
    executions = 0
 last_executed = 0
    denylisted = 0
   output_size =
     wall_time = 0
     user_time = 0
   system_time = 0
average_memory = 0
zwass

zwass

01/25/2022, 1:07 AM
Hmmm, the info in the
osquery_schedule
table is since the last startup. Persisting this across restarts is something @sharvil and I have been discussing putting on Fleet's osquery contribution roadmap.
h

HarlanF

01/25/2022, 1:08 AM
Ah, interesting, that's useful information, since we'd just rolled out updates today, and think osquery restarted most everywhere.
1:35 AM
Was poking around the source code, and found that I can do 
--schedule_lognames=true
flag and get the system to output which query it's executing to the logs.
s

seph

01/25/2022, 11:44 AM
IIRC you could also turn on verbose logging
h

HarlanF

01/25/2022, 9:33 PM
Yeah, I could, but across a fleet that'd be too much info. This'll get me what I need, the schedule_lognames thing above.