What is Logstash?
Logstash is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite “stash.” – elastic
In other words: Logstash is a software which can read data from multiple sources such as log files, tcp/udp port, MySQL (via Plugin) and MQTT (via Plugin) and then transforms the data into a desired format such as JSON with some augmented attributes and then sends the data into mainly Elasticsearch.
As you can see, Logstash is not given a complicated task to perform. All we expect from it is to stay alive and perform that task. But the setting up process and maintenance is such a pain. This reminds me of why I always refuse to use Java-based softwares.
Problem #1: Not starting with systemd inside an LXC container
No matter how much I tried, I never got it to start automatically with systemd. I had to open a screen session and then launch the process manually.
Ofcourse, whenever Logstash would crash, it would not autostart. And if i were to reboot the VM, i’d have to launch the process manually.
Problem #2: Long boot time
If I were to kill Logstash and relaunch it, it takes around 100 – 120 seconds. It’s that heavy. You lose 2 minutes worth of data. I guess a C++ based program would have taken a maximum of 10 seconds to do a similar task. Avoid Java-based programs.
Problem #3: Plugins (MQTT and MySQL)
If you want to use some other data sources or filters than provided, you’d have to compile the plugins yourself. When compiling the logstash-filter-jdbc_streaming plugin, I had to downgrade the my Java version from 11 to 8 else Logstash wouldn’t start.
And also the MQTT plugin would stop working randomly and crash the whole Logstash process which is unacceptable!
Problem #4: RAM Usage
Logstash would let me compile plugins without minimum of 2GB of RAM. I have managed to run Elasticsearch on just 256 MB of RAM. Why is Logstash so hungry? I ain’t gonna feed it anymore. Enough is enough.
Hello Node-Red
Node-Red already has a built in MQTT node. I had to install the elastic plugin via the interface itself and voila! I have a working Logstash alternative. Stay tuned for more on Node-Red
