Hello all I m wondering if anything is logged regarding the

Question

Hello all I m wondering if anything is logged regarding the specific metrics tripped when a query is denylisted We wrote some intensive queries hashing files recursively specifically to get them denylisted so that we could make sure we were catching those in our Splunk env Now that we have queries getting denied I want to understand specifically why they were denied and if that is logged somewhere that s going to be the quickest way Or is the recommended way to do this to run the profiling script

zwass · Answer

Schedule a query or live query to the `osquery schedule` table and look at the `denylisted` column Though I do see you say why they were denied and I don t have a good answer for that

Dan Achin · Answer

< zwass> that s how we are finding them we have a pack that runs `SELECT FROM osquery schedule WHERE denylisted= 1 `

Dan Achin · Answer

and we have all of this in Splunk I can see which hosts have denied queries and which queries were denied I was just hoping something might get logged by the watchdog that said denied over RAM allocation or something like that

zwass · Answer

There is some logging around that <https github com osquery osquery blob master osquery core watcher cpp L527 L537> but I m not sure it will tell you which query was executing

Dan Achin · Answer

Cool thanks for that

Dan Achin · Answer

We really wanted an easy way to tell if a query exceeded CPU or RAM allocations if denylisted Might have to start playing around with the profiler script I think since we aren t even logging locally we are currently only using the TLS logger Or we can add the filesystem logger plugin I guess need to discuss with the team Thanks again

Dan Achin · Answer

Actually < zwass> can you confirm if that watchdog logging should also be sent via the TLS logger or would it only write if filesystem was enabled

zwass · Answer

I do believe it should go to TLS but haven t looked at that in a while Let me know if you can confirm

Dan Achin · Answer

< zwass> will do We definitely haven t seen it in Splunk but let me look directly in Fleet logs a bit later

zwass · Answer

This would be in the status logs Are you also forwarding those to splunk

Dan Achin · Answer

yep

Dan Achin · Answer

and I looked at status logs for the hostnames uuids with denied queries and didn t see anything but maybe we have a field we need to extract

Dan Achin · Answer

well I m not finding anything in the logs when i grep for deny exceeded or limits

zwass · Answer

Hmm perhaps I m wrong on that front I could easily get nerd sniped into digging into this for the next hours but I have other things I gotta get to

Dan Achin · Answer

haha well I see nothing in staus log about it

Dan Achin · Answer

i think we ll enable the FS plugin on a couple hosts and see if we can find it there

Dan Achin · Answer

< zwass> I m thinking ensure the logging is written via TLS logger seems like it could be considered an issue or at least an enhancement Do you think it would get any traction if I opened a git issue on it

zwass · Answer

Yeah I think documenting it in an issue makes sense As far as traction not sure

Dan Achin · Answer

slightly smiling face

Dan Achin · Answer

ok thanks

Dan Achin · Answer

< zwass> just thought I d share what we found when looking into this deeper I had one of our guys that knows C C++ take a look at the code and it turns out that it does a syslog then immediately writes to the a WARNING log call to the osquery logging system So even though we haven t quite figured out why we don t see these in the var log osquery osquery WARNING logs on fleet we can still find the information we need in var log messages on the hosts that have denylisted queries so that s something Once we enable the filesystem plugin on some test hosts I m guessing we ll start to see the data in the local osquery WARNING files generated though I still think there needs to be a way to get these written to the tls logger as well

zwass · Answer

Thank you for the update I imagine this has something to do with the RocksDB connection Maybe the worker is holding that connection and the watcher is the one logging the reason

Dan Achin · Answer

yes I think it s definitely the parent process logging it i e the watcher

Dan Achin · Answer

also unfortunately this is right in the code ``` Since the watchdog cannot use the logger plugin the error message should be logged to stderr and to the system log ``` Looks like we might not be able to use tls to capture this but I m thinking the syslog plugin could be an option though that s going to increase our Splunk usage a lot

Dan Achin · Answer

< zwass> looks like I ll probably have to change my git issue to ask if the watcher can be updated to write to plugins