==Problem== 

A scheduled query of ours is only resulting in data some of the time

==Query details==

SELECT * FROM syslog WHERE facility = 'local0' AND severity = '6' AND tag LIKE 'foobar[%]';
- interval: 300
- removed: false

==Config details==

/etc/osquery/osquery.conf:

{
  "options": {
    "syslog_pipe_path": "/var/osquery/the_pipe",
    "config_plugin": "filesystem",
    "logger_plugin": "aws_kinesis,aws_firehose",
    "schedule_splay_percent": 10,
    "events_expiry": 3600,
    "verbose": false,
    "worker_threads": 2,
    "enable_monitor": false,
    "aws_kinesis_stream": "<snip>",
    "aws_firehose_stream": "<snip>",
    "aws_kinesis_random_partition_key": true,
    "syslog_events_expiry": 7200,
    "enable_syslog": true,
    "tls_server_certs": "<snip>",
    "aws_access_key_id": "<snip>",
    "aws_secret_access_key": "<snip>",
    "aws_region": "us-east-1"
  },
  "schedule": {
    "info": {
      "query": "SELECT * FROM osquery_info;",
      "interval": 86400
    }
  },
  "decorators": {
    "always": [
      "SELECT '<snip1>' as snip1",
      "SELECT '<snip2>' as snip2"
    ]
  },
  "packs": {
    "info": "/<snip>/packs/info.conf",
    "linux": "/<snip>/packs/linux.conf"
  }

/etc/rsyslog.d/osquery.conf:

template(
  name="OsqueryCsvFormat"
  type="string"
  string="%timestamp:::date-rfc3339,csv%,%hostname:::csv%,%syslogseverity:::csv%,%syslogfacility-text:::csv%,%syslogtag:::csv%,%msg:::csv%\n"
)
auth,authpriv,local2,cron.*,<http://local0.info|local0.info> action(type="ompipe" Pipe="/var/osquery/the_pipe" template="OsqueryCsvFormat")


==Debugging details==

- All of this checks out fine: <https://github.com/facebook/osquery/issues/1964>, which makes sense, because we get data some of the time

- 100% of the time the log is in /var/log/hourly

- Example log that was in /var/log/hourly, but didn't make it to Kinesis:

2017-03-10T04Z:2017-03-10T04:34:42.263152+00:00 the-hostname <http://local0.info|local0.info> foobar[12950]: - blah. blah: blah (blah blah)


==Guesses==

1)

"events_expiry": 3600,

"Timeout to expire eventing publish subscribe results from the backing-store. This expiration is only applied when results are queried. For example, if --events_expiry=1 then events will only practically exist for a single select from the subscriber. If no select occurs then events will be saved in the backing store indefinitely."

We have multiple queries that hit the `syslog` table, looking for different things.

It's unclear from the documentation if one syslog query could negatively impact another, aka result in some records being expunged?


2)

--syslog_events_max

"(default 100,000) sets a maximum number of logs to retain (oldest logs are deleted first if this number is surpassed).

Maybe we're exceeding 100k at various points, hence why we get the log sometimes?

3)

????