osqueryd is erroring out throwing following error for few of osquery #general

osqueryd is erroring out throwing following error ...

12/17/2021, 7:37 AM

osqueryd is erroring out throwing following error for few of the queries I am trying to run: ErrorLogTag":"read unix @-\u003e/var/osquery/sock/osquery.em: i/o timeout","level":"error" Is there anything that I need to change in config file? What can I do to resolve this error? worker_threads has been set to 4 currently. "worker_threads": "4"

seph

12/17/2021, 1:59 PM

I would expect that socket to be used to communicate to extensions. Are your queries using tables that come from an extension?

12/21/2021, 6:44 AM

This is happening for all tables. Like tables as basic as syslog_events

12/21/2021, 6:49 AM

journalctl logs for osqueryd shows the followiing log entries: 5914 eventsubscriberplugin.cpp:577] Removed 212 event batches (with 18446744073709551422 delete errors) for subscriber: syslog.syslog_events (limit: 100000, last query: never)

12/21/2021, 8:31 AM

journalctl logs error has been resolved by deleting the db and restarting the service

12/21/2021, 8:33 AM

But strangely, ErrorLogTag":"read unix @-\u003e/var/osquery/sock/osquery.em: i/o timeout","level":"error" is still showing up for some queries randomly. Like in one run, for some queries there is no error and for some queries this error creeps up. I'm unable to understand why!

12/21/2021, 8:51 AM

I see this majorly happening on syslog_events, last and process_open_sockets tables. And not on file_events, hardware_events and usb_devices tables. These are all the tables I am using so far.

12/21/2021, 8:52 AM

But a few seconds ago, even syslog_events, last and process_open_sockets tables queries ran and gave the desired outputs. But not anymore.

12/21/2021, 8:52 AM

Now syslog_events, last and process_open_sockets tables are only throwing ErrorLogTag":"read unix @-\u003e/var/osquery/sock/osquery.em: i/o timeout","level":"error"

seph

12/21/2021, 3:39 PM

Is this osqueryi? or osqueryd? Are you using any extensions? What options are you running with?

12/22/2021, 7:32 AM

This is osqueryd. Not getting this error when I ran with osqueryi. This is what osqueryd is using /usr/bin/osqueryd --flagfile /etc/osquery/osquery.flags --config_path /etc/osquery/osquery.conf I don't see any use of extensions though. This is how the flags file look. cat osquery.flags --extensions_socket=/var/osquery/sock/osquery.em --watchdog_memory_limit=1000

Divya

12/27/2021, 8:29 AM

@seph these are the options we use:

Divya

12/27/2021, 8:29 AM

Copy code

"host_identifier": "PLACEHOLDER_HOSTNAME",
"schedule_splay_percent": 10,
"config_plugin": "filesystem",
"logger_plugin": "filesystem",
"logger_path": "/var/log/osquery",
"disable_logging": "false",
"pidfile": "/var/osquery/osquery.pidfile",
"events_expiry": "1",
"database_path": "/var/osquery/osquery.db",
"worker_threads": "4",
"disable_events": "false",
"disable_audit": "false",
"audit_allow_config": "true",
"enable_syslog": "true",
"syslog_pipe_path": "/var/osquery/syslog_pipe",
"force": "true",
"audit_allow_sockets": "true",
"enable_file_events": "true",
"enable_ntfs_publisher": "true",
"schedule_default_interval": "3600"

Divya

12/27/2021, 8:31 AM

our queries run on a schedule and these errors are happeing when osquery is running on deamon mode. Strange part is, we are able to open the socket successfully

Divya

12/27/2021, 8:31 AM

but the query is giving these errors

seph

12/28/2021, 8:27 PM

I don't think I've seen this before. In another thread here, AP has talked about docker and other things. Is there anything unusual happening here?

Divya

12/31/2021, 7:23 AM

we trigger the queries from inside a docker container, but I believe that should not be an issue

Divya

12/31/2021, 7:25 AM

as the errors are random. How many parallel threads can osquery handle? Is that decided by "worker_threads": "4", ? Are there guidelines on setting this number?

6 Views

Open in Slack

Previous Next