{"id":38835,"date":"2016-07-31T13:10:47","date_gmt":"2016-07-31T07:40:47","guid":{"rendered":"http:\/\/www.tothenew.com\/blog\/?p=38835"},"modified":"2016-08-01T17:22:31","modified_gmt":"2016-08-01T11:52:31","slug":"making-elasticsearch-cluster-work-in-aws-mumbai-region","status":"publish","type":"post","link":"https:\/\/www.tothenew.com\/blog\/making-elasticsearch-cluster-work-in-aws-mumbai-region\/","title":{"rendered":"Elasticsearch: Making EC2-Discovery Work In AWS Mumbai Region"},"content":{"rendered":"<p>With the recently launched AWS Mumbai region, many of us might need to migrate our infrastructure there. In one such scenario, we have to setup Elasticsearch cluster on AWS EC2 instances and test it in Mumbai region.<\/p>\n<p><span style=\"font-weight: 400;\"><img decoding=\"async\" loading=\"lazy\" class=\"alignnone  wp-image-38845\" src=\"\/blog\/wp-ttn-blog\/uploads\/2016\/07\/aws-square.png\" alt=\"aws-square\" width=\"117\" height=\"117\" \/>\u00a0<img decoding=\"async\" loading=\"lazy\" class=\"alignnone  wp-image-38844\" src=\"\/blog\/wp-ttn-blog\/uploads\/2016\/07\/6764390.png\" alt=\"6764390\" width=\"119\" height=\"119\" \/>\u00a0<\/span><\/p>\n<p>While <a title=\"aws migration services\" href=\"http:\/\/www.tothenew.com\/devops-automation-consulting\">migrating Elasticsearch cluster from AWS<\/a> Singapore region to Aws Mumbai region, we faced few challenges, one of them is described below:<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-38847\" src=\"\/blog\/wp-ttn-blog\/uploads\/2016\/07\/information-icon.png\" alt=\"information-icon\" width=\"12\" height=\"12\" \/><strong>\u00a0 Challenge<\/strong>:\u00a0<span style=\"font-weight: 400;\">There is a plugin which enables Elasticsearch cluster to discover new nodes automatically and added them to the cluster called \u00a0&#8220;<a href=\"https:\/\/www.elastic.co\/guide\/en\/elasticsearch\/plugins\/2.3\/cloud-aws.html\">AWS<\/a><a href=\"https:\/\/www.elastic.co\/guide\/en\/elasticsearch\/plugins\/2.3\/cloud-aws.html\"> Cloud<\/a><a href=\"https:\/\/www.elastic.co\/guide\/en\/elasticsearch\/plugins\/2.3\/cloud-aws.html\"> Plugin<\/a><strong>&#8220;<\/strong><\/span><span style=\"font-weight: 400;\">. The plugin used EC2 instance tags and region to add nodes to the cluster. Once we have installed the plugin, we need to configure the &#8220;elasticsearch.yml&#8221; file. The plugin segment should look similar to this:<\/span><\/p>\n<p>[js] plugin.mandatory: cloud-aws<br \/>\n cluster.name: mynewcluster<br \/>\n cloud.aws.access_key: mykey<br \/>\n cloud.aws.secret_key: mysecret<br \/>\n cloud.aws.region: us-east-1<br \/>\n discovery.type: ec2<br \/>\n discovery.ec2.tag.elasticsearch: mynewcluster[\/js]<\/p>\n<p>Here, we updated the cloud.aws.region to \u2018ap-south-1\u2019 (Mumbai) and restarted our Elasticsearch nodes but the nodes could not get added to the cluster and there were errors which lead us to conclude that the <b>plugin does not support Mumbai region yet<\/b><span style=\"font-weight: 400;\">.<\/span><\/p>\n<p><b><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-38846\" src=\"\/blog\/wp-ttn-blog\/uploads\/2016\/07\/download.jpg\" alt=\"download\" width=\"19\" height=\"15\" \/>\u00a0The Solution:\u00a0<\/b>Download\/Clone the plugin and get ready to make some changes. Basically, we want to update the source files and add support for Mumbai region. Once we have downloaded the plugin, we will search for the files that contain AWS Regions in them. The desired file could be searched as shown below:<\/p>\n<p>[js]$ grep -rnw .\/ -e &#8216;ap-southeast-1&#8242;[\/js]<\/p>\n<p><span style=\"font-weight: 400;\">We will get all the filenames and paths that contain region names and now we know where to make changes. To be precise there are three files:<\/span><\/p>\n<ul>\n<li><i><span style=\"font-weight: 400;\">S3Repository.java <\/span><\/i><\/li>\n<li><i><span style=\"font-weight: 400;\">InternalAwsS3Service.java<\/span><\/i><\/li>\n<li><i><span style=\"font-weight: 400;\">AwsEc2Service.java<\/span><\/i><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">There is a segment of code in all these files where \u2018if\u2019 operator checks for the Region and Endpoint and it is where all the Regions and Endpoints are to be updated. Just add one more entry here for Mumbai region (ap-south-1) in all the files:<\/span><\/p>\n<p>[js]else if (region.equals(&#8216;ap-south&#8217;) || region.equals(&#8216;ap-south-1&#8217;)) {<br \/>\nendpoint = &#8216;ec2.ap-south-1.amazonaws.com&#8217;;<br \/>\n}[\/js]<\/p>\n<p>That is it!<\/p>\n<p>We can now use Jenkins to do a Maven build on the plugin\u2019s local git repository. Once done, we need to replace the old java files with newer ones in the existing plugin on each Elasticsearch node.<\/p>\n<p><span style=\"font-weight: 400;\">On restarting Elasticsearch we will see that the new nodes are getting \u2018discovered\u2019 in the Region and are getting added to the cluster. Things work fine for Mumbai region just like any other. <\/span><\/p>\n<p><span style=\"font-weight: 400;\">See you with another write-up!<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>With the recently launched AWS Mumbai region, many of us might need to migrate our infrastructure there. In one such scenario, we have to setup Elasticsearch cluster on AWS EC2 instances and test it in Mumbai region. \u00a0\u00a0 While migrating Elasticsearch cluster from AWS Singapore region to Aws Mumbai region, we faced few challenges, one [&hellip;]<\/p>\n","protected":false},"author":934,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"status","meta":{"iawp_total_views":42},"categories":[1174,2348,1],"tags":[1547,3871,3874,4843,3873,1524,3872,1252,3356,332],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/38835"}],"collection":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/users\/934"}],"replies":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/comments?post=38835"}],"version-history":[{"count":0,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/38835\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/media?parent=38835"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/categories?post=38835"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/tags?post=38835"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}