I am trying to implement YARA scanning through osquery (malware_removal extension) on Kolide Fleet. For a directory with relatively large number of files, it's showing me the error 'Memory limits exceeded'. Can someone help me with this?

osqueryd

 uses a watchdog process to monitor the memory and CPU utilization of threads executing the query schedule. If any performance limit is violated, the “worker” process will be restarted. The default threshold per the documentation is 200MB. I would increase this threshold with the following Osquery flag:

--watchdog_memory_limit=X

https://osquery.readthedocs.io/en/stable/installation/cli-flags/

Increasing the watchdog memory limit worked for me. But everytime I am increasing the directory size, I have to increase the memory limit. Is there any limit which I can be sure of that it will do the job?

I assume that is because Osquery is reading all of the contents within a directory into memory to scan the contents. If that is true, it might be a bug because it should be only loading one item at a time.