I've been struggling with this for a few days and I've got to the point where I'm hoping someone else can guide me...
I've installed kolide fleet, elasticsearch, kibana, and i've installed logstash all on the same server. Everything is fine (kolide fleet, elasticsearch, kibana are all running) and my server is humming along at under 10% utilized, but as soon as I try starting logstash my server becomes close to 100% utilized and very sluggish, and while logstash has created /var/log/logstash, it never writes anything in there so I have no logs to check to see what is going on. I suspect there is a problem with my logstash configuration and maybe that is causing it, as I've not really been able to get a handle on how the configuration should be done - most people it seems use filebeat to send the osquery logs to logstash to send them to elasticsearch, but I read that filebeat is not necessary if you are using logstash on the same machine as the osquery logs are being stored on. So this is my logstash configuration file (/etc/logstash/conf.d/47-osquery-output.conf):
input {
file {
path => "/var/log/osquery/osqueryd.results.log"
type => "osquery_json"
codec => "json"
}
}
filter {
if [type] == "osquery_json" {
date {
match => [ "unixTime", "UNIX" ]
}
}
}
output {
stdout {}
elasticsearch {
hosts=> "127.0.0.1:9200"
}
}
Can anyone suggest why logstash is overloading my server? and why it is not creating any logs?
can anyone confirm I've got a correct configuration in this conf file? /var/log/osquery/osqueryd.results.log is the file being output from kolide fleet. I notice that it is owned by root and in the root group but it is read for owner, group and other so it shouldn't be a prolem. and elasticsearch is listening on 127.0.0.1:9200. I don't understand what the filter in the middle of the config is for though - is it really necessary? what does it do?
Thanks in advance!