Title
#general
a

Andrew Wei

03/03/2020, 1:54 AM
hello, has anyone had to deal with the watchdog constantly killing your osquery worker?
1:55 AM
sudo osqueryd --verbose
I0303 01:48:58.604156 27155 init.cpp:418] osquery initialized [version=4.1.2]
I0303 01:48:58.617558 27155 system.cpp:330] Found stale process for osqueryd (26310)
I0303 01:48:58.617607 27155 system.cpp:362] Writing osqueryd pid (27155) to /var/run/osqueryd.pidfile
I0303 01:48:58.617666 27155 extensions.cpp:349] Could not autoload extensions: Failed reading: /etc/osquery/extensions.load
I0303 01:48:58.618319 27156 watcher.cpp:583] osqueryd watcher (27155) executing worker (27157)
I0303 01:48:58.627338 27157 init.cpp:415] osquery worker initialized [watcher=27155]
I0303 01:48:58.627902 27157 rocksdb.cpp:131] Opening RocksDB handle: /var/osquery/osquery.db
I0303 01:48:59.130722 27157 auto_constructed_tables.cpp:93] Removing stale ATC entries
I0303 01:48:59.131106 27165 interface.cpp:268] Extension manager service starting: /var/osquery/osquery.em
I0303 01:49:00.211717 27157 aws_util.cpp:469] Using AWS region from flag: us-east-1
I0303 01:49:00.211901 27157 aws_kinesis.cpp:79] Kinesis logging initialized with stream: kinesis_metrics_osquery
I0303 01:49:00.224510 27157 events.cpp:863] Event publisher not enabled: syslog: Publisher disabled via configuration
I0303 01:49:01.261039 27166 aws_log_forwarder.h:214] aws_kinesis: Successfully sent 6 out of 6 log records
I0303 01:49:12.312868 27166 aws_log_forwarder.h:214] aws_kinesis: Successfully sent 6 out of 6 log records
W0303 01:49:58.632416 27156 watcher.cpp:331] osqueryd worker (27157) stopping: Maximum sustainable CPU utilization limit exceeded: 60
I0303 01:49:59.033196 27156 watcher.cpp:583] osqueryd watcher (27155) executing worker (27539)
I0303 01:49:59.042430 27539 init.cpp:415] osquery worker initialized [watcher=27155]
I0303 01:49:59.042579 27539 rocksdb.cpp:131] Opening RocksDB handle: /var/osquery/osquery.db
I0303 01:49:59.546882 27539 auto_constructed_tables.cpp:93] Removing stale ATC entries
I0303 01:49:59.547086 27547 interface.cpp:268] Extension manager service starting: /var/osquery/osquery.em
I0303 01:50:00.647491 27539 aws_util.cpp:469] Using AWS region from flag: us-east-1
I0303 01:50:00.647614 27539 aws_kinesis.cpp:79] Kinesis logging initialized with stream: kinesis_metrics_osquery
I0303 01:50:00.661666 27539 events.cpp:863] Event publisher not enabled: syslog: Publisher disabled via configuration
W0303 01:50:11.036162 27156 watcher.cpp:331] osqueryd worker (27539) stopping: Maximum sustainable CPU utilization limit exceeded: 12
I0303 01:50:11.436947 27156 watcher.cpp:583] osqueryd watcher (27155) executing worker (27630)
I0303 01:50:11.446023 27630 init.cpp:415] osquery worker initialized [watcher=27155]
I0303 01:50:11.446558 27630 rocksdb.cpp:131] Opening RocksDB handle: /var/osquery/osquery.db
I0303 01:50:11.940778 27630 auto_constructed_tables.cpp:93] Removing stale ATC entries
I0303 01:50:11.940834 27638 interface.cpp:268] Extension manager service starting: /var/osquery/osquery.em
I0303 01:50:13.001374 27630 aws_util.cpp:469] Using AWS region from flag: us-east-1
I0303 01:50:13.001509 27630 aws_kinesis.cpp:79] Kinesis logging initialized with stream: kinesis_metrics_osquery
I0303 01:50:13.031401 27630 events.cpp:863] Event publisher not enabled: syslog: Publisher disabled via configuration
I0303 01:50:14.070098 27639 aws_log_forwarder.h:214] aws_kinesis: Successfully sent 6 out of 6 log records
W0303 01:50:23.439877 27156 watcher.cpp:331] osqueryd worker (27630) stopping: Maximum sustainable CPU utilization limit exceeded: 12
I0303 01:50:23.840651 27156 watcher.cpp:583] osqueryd watcher (27155) executing worker (27723)
1:56 AM
osquery is getting killed off hundreds/thousands of times a day, with this log:
osqueryd worker (27630) stopping: Maximum sustainable CPU utilization limit exceeded: 12
s

seph

03/03/2020, 1:58 AM
osquery consumes CPU based on what’s it’s doing. So, presumably, your query configuration exceeds what you’ve told the watchdog to allow. Change your queries, and/or up your limit
a

Andrew Wei

03/03/2020, 2:00 AM
i just did a search and found this piece of advice from @sundsta :
I saw similar when the local database was corrupted. Stopping the service, deleting the local 
osquery.db
, and restarting the service resolved the issue
2:01 AM
looks like it's working so far 🤞