Title
#kolide
s

Stephen

03/13/2019, 5:07 AM
I've been struggling with this for a few days and I've got to the point where I'm hoping someone else can guide me... I've installed kolide fleet, elasticsearch, kibana, and i've installed logstash all on the same server. Everything is fine (kolide fleet, elasticsearch, kibana are all running) and my server is humming along at under 10% utilized, but as soon as I try starting logstash my server becomes close to 100% utilized and very sluggish, and while logstash has created /var/log/logstash, it never writes anything in there so I have no logs to check to see what is going on. I suspect there is a problem with my logstash configuration and maybe that is causing it, as I've not really been able to get a handle on how the configuration should be done - most people it seems use filebeat to send the osquery logs to logstash to send them to elasticsearch, but I read that filebeat is not necessary if you are using logstash on the same machine as the osquery logs are being stored on. So this is my logstash configuration file (/etc/logstash/conf.d/47-osquery-output.conf): input { file { path => "/var/log/osquery/osqueryd.results.log" type => "osquery_json" codec => "json" } } filter { if [type] == "osquery_json" { date { match => [ "unixTime", "UNIX" ] } } } output { stdout {} elasticsearch { hosts=> "127.0.0.1:9200" } } Can anyone suggest why logstash is overloading my server? and why it is not creating any logs? can anyone confirm I've got a correct configuration in this conf file? /var/log/osquery/osqueryd.results.log is the file being output from kolide fleet. I notice that it is owned by root and in the root group but it is read for owner, group and other so it shouldn't be a prolem. and elasticsearch is listening on 127.0.0.1:9200. I don't understand what the filter in the middle of the config is for though - is it really necessary? what does it do? Thanks in advance!
12:30 PM
So I just discovered this in syslog. I'm not yet sure what it might mean but I share it in case it confirms someone's suspicions about the source of my problem:
s

seph

03/13/2019, 1:57 PM
Have you done any initial debugging? Is a process consuming a lot of CPU? A lot of RAM? What does 100% utilized mean?
s

Stephen

03/13/2019, 3:19 PM
I don't know where else to look. All I know is that until I start logstash the server is running perfectly and then it goes to near 100%. I'm taking these numbers from the linux top command
3:19 PM
it is cpu utilization from what I can tell
3:19 PM
thanks for your reply by the way Seph
s

seph

03/13/2019, 5:09 PM
Knowing you ran
top
and it shows logstash is consuming a lot of CPU is valuable. That wasn’t clear from your initial comment.
5:10 PM
I don’t know much about logstash. Sorry
s

Stephen

03/14/2019, 5:50 AM
Thanks for helping clarify Seph. That's the weird thing. It doesn't show logstash is consuming a lot of resources, I see some strange "shim" processes and other random things but not logstash. But the high utilization only occurs when I start logstash. I was also thinking more about this overnight and realised that I did do some other troubleshooting last week that I didn't think was relevant at the time but now with the syslog results mentioning trouble talking to elasticsearch it seems more relevant: ubuntu@ip-172-30-0-162:~$ curl http://127.0.0.1:9200 { "name" : "3sHDhp8", "cluster_name" : "elasticsearch", "cluster_uuid" : "_RRhXzMkQRi2cl9ree9S5Q", "version" : { "number" : "6.6.1", "build_flavor" : "default", "build_type" : "deb", "build_hash" : "1fd8f69", "build_date" : "2019-02-13T17🔟04.160291Z", "build_snapshot" : false, "lucene_version" : "7.6.0", "minimum_wire_compatibility_version" : "5.6.0", "minimum_index_compatibility_version" : "5.0.0" }, "tagline" : "You Know, for Search" } This tells me elastic search is actually listening
10:04 AM
So I'm guessing this is not really a kolide - logstash issue but a logstash elastic search issue so I'm going to post in their community instead. Thanks for reading though
s

seph

03/14/2019, 1:13 PM
what does top show?
s

Stephen

03/14/2019, 6:07 PM
I'm not in a position to bring the server to a grinding halt again until Saturday so I'm afraid I can't tell you at the moment. However, over in the logstash community it seems to be that there is a configuration error in the logstash configuration file that tells it how to pick up the files from osquery/kolide and this is causing logstash to go into an endless restarting cycle. Apparently the JVM that it runs in starts running very heavily in such a circumstance. So if anyone here has experience in configuring logstash to pickup osquery logs on the same machine directly, rather than unnecessarily using filebeat to pick them up from the same machine and shuffle them to logstash which logstash can apparently do itself when it is on the same machine, I'd appreciate any advice on how to correct the configuration.
12:05 PM
well as a final update to this, it seems that logstash uses an inordinate amount of RAM and the jvm that runs it was unsuccessfully trying to allocate it. Even with no configuration and just logstash -help it couldn't even display help because it was too busy trying to get more ram than it had any right too 🙂 so I've abandoned it in favour of ingest nodes.