![\"\"](\"https:\/\/nayarweb.com\/blog\/wp-content\/uploads\/2017\/05\/Spectacle.TJ4438.png\")
<\/p>\nI had this 1 Elasticsearch node that had higher load than his colleagues.<\/p>\n
<\/p>\n
Previously, they were all living in perfect harmony<\/p>\n
![\"\"](\"https:\/\/nayarweb.com\/blog\/wp-content\/uploads\/2017\/05\/Spectacle.kn4438.png\")
<\/p>\n
It all started when I rebooted the hypervisor the elastic node was on.<\/p>\n
I looked in the KVM virsh files to see if the node had differences with the others. I noticed only this node wasn’t using the `virtio` driver for network and disk. I changed from `ide` and `e1000` driver to `virtio` for disk and network respectively. Rebooted the node but still couldn’t match the performance of his counterparts.<\/p>\n
This problem had to be solved because the ElasticSearch cluster performance is directly affected by the slowest node in the cluster. If a node is slow, it’s better it’s not in the cluster. The 75h percentile requests was more than 1.5s. Usually it was around 400ms in peak hours. My 99.9th percentile exceeded 50 seconds. It was really dangerous. The cluster receives 1 million documents per minute.<\/p>\n
![\"\"](\"https:\/\/nayarweb.com\/blog\/wp-content\/uploads\/2017\/05\/Spectacle.TJ4498.png\")
<\/p>\n
`iotop -a` showed the same processes running but had high IO on `[jbd2\/vdb-8]`. It just confirmed our problem. But no solution as of yet.<\/p>\n
I noticed on the network graph that the node was not able to send more than 600MB per 5 mins at all times when previously it could.<\/p>\n
![\"\"](\"https:\/\/nayarweb.com\/blog\/wp-content\/uploads\/2017\/05\/Spectacle.dT4498-1024x199.png\")
<\/p>\n
There must be some kind of restriction on network. It must be when the hypervisor rebooted, the network negotiation had some issues. Comparing values from 2 hypervisors confirmed the hypothesis<\/p>\n
root@hypervisor0:~# mii-tool eth0\r\neth0: negotiated 100baseTx-FD, link ok\r\n\r\nroot@hypervisor1:~# mii-tool eth0\r\neth0: negotiated 1000baseT-FD flow-control, link ok\r\n<\/pre>\nWe can see the speed difference is major here. The VM reported high disk IO because Ceph relies on network to read\/write data.<\/p>\n
\nMonitor your cluster farm with Collectiva (Beta)<\/span><\/a>, a product of nayarweb.com<\/h3>\n<\/div>\n","protected":false},"excerpt":{"rendered":"
I had this 1 Elasticsearch node that had higher load than his colleagues. Previously, they were all living in perfect harmony It all started when I rebooted the hypervisor the elastic node was on. I looked in the KVM virsh files to see if the node had differences with the others. I noticed only this … Continue reading High Load on one of ElasticSearch Node on Ceph<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[210],"tags":[230,225,209],"class_list":["post-3604","post","type-post","status-publish","format-standard","hentry","category-technology","tag-ceph","tag-elasticsearch","tag-virtualisation"],"_links":{"self":[{"href":"https:\/\/nayarweb.com\/blog\/wp-json\/wp\/v2\/posts\/3604","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nayarweb.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nayarweb.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nayarweb.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nayarweb.com\/blog\/wp-json\/wp\/v2\/comments?post=3604"}],"version-history":[{"count":2,"href":"https:\/\/nayarweb.com\/blog\/wp-json\/wp\/v2\/posts\/3604\/revisions"}],"predecessor-version":[{"id":3610,"href":"https:\/\/nayarweb.com\/blog\/wp-json\/wp\/v2\/posts\/3604\/revisions\/3610"}],"wp:attachment":[{"href":"https:\/\/nayarweb.com\/blog\/wp-json\/wp\/v2\/media?parent=3604"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nayarweb.com\/blog\/wp-json\/wp\/v2\/categories?post=3604"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nayarweb.com\/blog\/wp-json\/wp\/v2\/tags?post=3604"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}