If you want to set the watchdog memory limit, you have to pass to osqueryd the flag

--watchdog_memory_limit=1000