Why you should Node-RED instead of Java-based Logstash

What is Logstash?

Logstash is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite “stash.” – elastic

In other words: Logstash is a software which can read data from multiple sources such as log files, tcp/udp port, MySQL (via Plugin) and MQTT (via Plugin) and then transforms the data into a desired format such as JSON with some augmented attributes and then sends the data into mainly Elasticsearch.

As you can see, Logstash is not given a complicated task to perform. All we expect from it is to stay alive and perform that task. But the setting up process and maintenance is such a pain. This reminds me of why I always refuse to use Java-based softwares.

Problem #1: Not starting with systemd inside an LXC container

No matter how much I tried, I never got it to start automatically with systemd. I had to open a screen session and then launch the process manually.

Ofcourse, whenever Logstash would crash, it would not autostart. And if i were to reboot the VM, i’d have to launch the process manually.

Problem #2: Long boot time

If I were to kill Logstash and relaunch it, it takes around 100 – 120 seconds. It’s that heavy. You lose 2 minutes worth of data. I guess a C++ based program would have taken a maximum of 10 seconds to do a similar task. Avoid Java-based programs.

Problem #3: Plugins (MQTT and MySQL)

If you want to use some other data sources or filters than provided, you’d have to compile the plugins yourself. When compiling the logstash-filter-jdbc_streaming plugin, I had to downgrade the my Java version from 11 to 8 else Logstash wouldn’t start.

And also the MQTT plugin would stop working randomly and crash the whole Logstash process which is unacceptable!

Problem #4: RAM Usage

Logstash would let me compile plugins without minimum of 2GB of RAM. I have managed to run Elasticsearch on just 256 MB of RAM. Why is Logstash so hungry? I ain’t gonna feed it anymore. Enough is enough.

Hello Node-Red

Node-Red already has a built in MQTT node. I had to install the elastic plugin via the interface itself and voila! I have a working Logstash alternative. Stay tuned for more on Node-Red

 

Leave a Reply

Your email address will not be published.