Author: Nayar

Published on by Nayar

High Load on one of ElasticSearch Node on Ceph

I had this 1 Elasticsearch node that had higher load than his colleagues.

Previously, they were all living in perfect harmony

It all started when I rebooted the hypervisor the elastic node was on.

I looked in the KVM virsh files to see if the node had differences with the others. I noticed only this node wasn’t using the `virtio` driver for network and disk. I changed from `ide` and `e1000` driver to `virtio` for disk and network respectively. Rebooted the node but still couldn’t match the performance of his counterparts.

This problem had to be solved because the ElasticSearch cluster performance is directly affected by the slowest node in the cluster. If a node is slow, it’s better it’s not in the cluster. The 75h percentile requests was more than 1.5s. Usually it was around 400ms in peak hours. My 99.9th percentile exceeded 50 seconds. It was really dangerous. The cluster receives 1 million documents per minute.

`iotop -a` showed the same processes running but had high IO on `[jbd2/vdb-8]`. It just confirmed our problem. But no solution as of yet.

I noticed on the network graph that the node was not able to send more than 600MB per 5 mins at all times when previously it could.

There must be some kind of restriction on network. It must be when the hypervisor rebooted, the network negotiation had some issues. Comparing values from 2 hypervisors confirmed the hypothesis

root@hypervisor0:~# mii-tool eth0
eth0: negotiated 100baseTx-FD, link ok

root@hypervisor1:~# mii-tool eth0
eth0: negotiated 1000baseT-FD flow-control, link ok

We can see the speed difference is major here. The VM reported high disk IO because Ceph relies on network to read/write data.

Monitor your cluster farm with Collectiva (Beta), a product of nayarweb.com

Published on by Nayar

Mauritians won’t be silent!

Mauritius is going through a very difficult phase. Lots of time we hear that just posting on Facebook won’t change a thing. Mauritians just talk, no action. Mauritians are cowards.
 
The prophet of Islam reportedly said:
 
“Whosoever of you sees an evil, let him change it with his hand; and if he is not able to do so, then [let him change it] with his tongue; and if he is not able to do so, then with his heart — and that is the weakest of faith.” [Muslim]
 
We cannot actually stop a minister/ti-copin from stealing. We don’t even have access to his office. Forget even the papers. We don’t know what’s happening there.
 
Our voice is what we common Mauritians have — at-least for now. We can see bad things happening. We should talk about it.
 
I don’t have a fucking clue what ICAC et al. do. They are supposed to have powerful hands to stop corruption. Maybe they are helpless too. If your are helpless ICAC, voice it out! We Mauritians are paying you big money.
 
If or When (i hope not) Mauritius becomes like North Korea, we’ll put our votes in ballots in favour of Government despite the heart says otherwise because it then becomes a question of survival.
 
I hope Mauritius never becomes like North Korea. Mauritius is a beautiful country with amazing people. We can’t allow politicians to ruin our reputation and divide us.
 
Peace.
 
 
Published on by Nayar

Reading Files Line by Line in Python

You write a script in python which has to read a file, loop through each line to do some work. Seems easy right?

f = open('loulou.txt', 'r')
lines = f.readlines()
for line in lines:
	print line

Problem is when your file becomes big, let’s say 1500MB, the code will still run on your development laptop but it will fail on a server on cloud with 1GB of RAM. You’ll have an Out Of Memory error which can lead to some very important process being killed if OOM priorities are not mastered.

You can use the Collectiva service by NAYARWEB.COM to get alerted near-realtime whenever Out of Memory occurs on one of your servers.

To fix the code above, you can read the file line by line from the disk:

f = open('loulou.txt', 'r')
for line in f:
        print line

Is this always better? No. The current version will run slightly slower than the first code depending on your storage backend.

It’s amazing how little differences like these can have a huge impact on production servers.

Happy Monitoring to all System Admins out there 🙂

Published on by Nayar

Car Not Start on Rainy Days

If I drove VW Polo everyday, it would start perfect and run perfect for months. But if I let the car sit for more than 24 hours, on the next day it would have lots of trouble to start the engine.

I’d have to wait for the sun to warm it a little and then It would turn on as if nothing was ever wrong with it.

The Cause:

It turns out that the rotor in the distributor had a huge hole in it.

So my dad cut a tiny piece of wire and put it inside the hole linking the 2 extremities of the hole. I think the inner stuff is graphite. He put it back in the car. And lo! It started despite the heavy rain! 😀

The Solution

But we decided to buy a new rotor in a car shop nearby for Rs 300. My car’s been great since. Noticed some performance improvement.

Take car of your car. Drive safe.

Published on by Nayar

Abolition of Re-registration Duty for Cars in Mauritius

This morning on Facebook, I came across someone who posted the following:

Let’s have a look on NTA’s instructions on how to sell your car.

source: nta.govmu.org [accessed: 02/05/2017]
The procedures seems fair and easy and not costly. However, the “The deed of sale should be registered at the Registrar General’s Department” part will be very expensive. Why doesn’t the NTA put the all the cost in one place? Anyways, there are no direct links from that page to know how much you have to pay to register the deed of sale.

I had to google it.

The second link is the PDF which has the informations.

2 months ago, I wanted to buy a newer car. But with these exorbitant duties, I refrained.

https://www.facebook.com/NayarJoolfoo/posts/10210709586528792

Now we ask ourselves, how many old cars are involved in fatal accidents? Every time I watch the news, I see BMW, Mercedes, etc being crushed. Where are the old cars? Why are they so hated?

 

Older cars are used exclusively by poor people. Each time a poor person buys an old car, he has to pay Rs 6,500, Rs 10,400, Rs 13,000 for >1001cc, >1251cc, 1501cc respectively.

Math question for CPE students and adults too. [300 points]

Let’s assume each time an old car is sold among the poor, Rs 10,000 goes to the government. How many poor people it takes to pay for the new Pravind Jugnauth car which is valued at Rs 23,541,673? Please leave the answer in the comments.