{"id":4312,"date":"2019-08-09T14:08:25","date_gmt":"2019-08-09T10:08:25","guid":{"rendered":"https:\/\/nayarweb.com\/blog\/?p=4312"},"modified":"2019-08-09T14:08:25","modified_gmt":"2019-08-09T10:08:25","slug":"why-you-should-node-red-instead-of-java-based-logstash","status":"publish","type":"post","link":"https:\/\/nayarweb.com\/blog\/2019\/why-you-should-node-red-instead-of-java-based-logstash\/","title":{"rendered":"Why you should Node-RED instead of Java-based Logstash"},"content":{"rendered":"

What is Logstash?<\/h2>\n

Logstash is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite “stash.” – elastic<\/p><\/blockquote>\n

In other words:<\/strong> Logstash is a software which can read data from multiple sources such as log files, tcp\/udp port, MySQL (via Plugin) and MQTT (via Plugin) and then transforms the data into a desired format such as JSON with some augmented attributes and then sends the data into mainly Elasticsearch.<\/p>\n

\"\"<\/p>\n

As you can see, Logstash is not given a complicated task to perform. All we expect from it is to stay alive and perform that task. But the setting up process and maintenance is such a pain. This reminds me of why I always refuse to use Java-based softwares.<\/p>\n

Problem #1: Not starting with systemd inside an LXC container<\/h2>\n

No matter how much I tried, I never got it to start automatically with systemd. I had to open a screen session and then launch the process manually.<\/p>\n

Ofcourse, whenever Logstash would crash, it would not autostart. And if i were to reboot the VM, i’d have to launch the process manually.<\/p>\n

Problem #2: Long boot time<\/h2>\n

If I were to kill Logstash and relaunch it, it takes around 100 – 120 seconds. It’s that heavy. You lose 2 minutes worth of data. I guess a C++ based program would have taken a maximum of 10 seconds to do a similar task. Avoid Java-based programs.<\/p>\n

\"\"<\/a><\/p>\n

\"\"<\/a><\/p>\n

Problem #3: Plugins (MQTT and MySQL)<\/h2>\n

If you want to use some other data sources or filters than provided, you’d have to compile the plugins yourself. When compiling the logstash-filter-jdbc_streaming<\/a> plugin, I had to downgrade the my Java version from 11 to 8 else Logstash wouldn’t start.<\/p>\n

And also the MQTT plugin<\/a> would stop working randomly and crash the whole Logstash process which is unacceptable!<\/p>\n

Problem #4: RAM Usage<\/h2>\n

Logstash would let me compile plugins without minimum of 2GB of RAM. I have managed to run Elasticsearch on just 256 MB of RAM. Why is Logstash so hungry? I ain’t gonna feed it anymore. Enough is enough.<\/p>\n

Hello Node-Red<\/h2>\n

Node-Red already has a built in MQTT node. I had to install the elastic plugin via the interface itself and voila! I have a working Logstash alternative. Stay tuned for more on Node-Red<\/p>\n

\"\"<\/a><\/p>\n

 <\/p>\n","protected":false},"excerpt":{"rendered":"

What is Logstash? Logstash is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite “stash.” – elastic In other words: Logstash is a software which can read data from multiple sources such as log files, tcp\/udp port, MySQL (via … Continue reading Why you should Node-RED instead of Java-based Logstash<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[210],"tags":[225,232],"class_list":["post-4312","post","type-post","status-publish","format-standard","hentry","category-technology","tag-elasticsearch","tag-internet-of-things-iot"],"_links":{"self":[{"href":"https:\/\/nayarweb.com\/blog\/wp-json\/wp\/v2\/posts\/4312","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nayarweb.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nayarweb.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nayarweb.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nayarweb.com\/blog\/wp-json\/wp\/v2\/comments?post=4312"}],"version-history":[{"count":6,"href":"https:\/\/nayarweb.com\/blog\/wp-json\/wp\/v2\/posts\/4312\/revisions"}],"predecessor-version":[{"id":4326,"href":"https:\/\/nayarweb.com\/blog\/wp-json\/wp\/v2\/posts\/4312\/revisions\/4326"}],"wp:attachment":[{"href":"https:\/\/nayarweb.com\/blog\/wp-json\/wp\/v2\/media?parent=4312"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nayarweb.com\/blog\/wp-json\/wp\/v2\/categories?post=4312"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nayarweb.com\/blog\/wp-json\/wp\/v2\/tags?post=4312"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}