{"id":34213,"date":"2016-05-04T10:35:08","date_gmt":"2016-05-04T05:05:08","guid":{"rendered":"http:\/\/www.tothenew.com\/blog\/?p=34213"},"modified":"2024-01-02T17:47:22","modified_gmt":"2024-01-02T12:17:22","slug":"elasticsearch-migration-found-to-aws-ec2","status":"publish","type":"post","link":"https:\/\/www.tothenew.com\/blog\/elasticsearch-migration-found-to-aws-ec2\/","title":{"rendered":"Elasticsearch Migration : Found to AWS EC2"},"content":{"rendered":"

Our DevOps team<\/a> was using Found for one of our projects in the production environment. We have been facing a\u00a0problem with found where it\u2019s memory pressure frequently goes up and does not drop down so easily and until the time it remains up the Found was not able to serve\u00a0the requests. Then, we decided to move to self-hosted Elasticsearch cluster.<\/p>\n

\"\"<\/p>\n

Scenario<\/strong>: Migration of Elasticsearch cluster from Found to AWS EC2 server<\/p>\n

Below is the step by step procedure to migrate Elasticsearch cluster from Found to AWS EC2:<\/strong><\/p>\n

    \n
  1. Stop the indexing requests of the data on Found.<\/li>\n
  2. Take the\u00a0dump of data from Found to AWS S3 using \u201c_snapshot\u201d API provided by Elasticsearch.\n

    [js]#!bin\/bash
    \ncurl -u : -XPUT ":9200\/_snapshot\/backup" -d’
    \n{
    \n"type": "s3",
    \n"settings": {
    \n"bucket": "bucket-name",
    \n"region": "ap-southeast-1",
    \n"access_key": "——————————–",
    \n"secret_key": "—————————————————",
    \n"compress": true,
    \n"base_path": "directory_name_inside_the_bucket"
    \n}
    \n}’;
    \ncurl -u : -XPUT ":9200\/_snapshot\/backup\/snapshot_prod?wait_for_completion=true";[\/js]<\/p>\n

    The first curl request would register the repository \u201cbackup\u201d on the Found where we are telling the Found that this repository would take backup on s3. We would have to provide s3 information in the curl request:
    \nbucket :- Name of S3 bucket where we want to take backup.
    \nregion :- Name of the region of the s3 bucket.
    \naccess_key :- Access key of AWS account where the bucket is present.
    \nsecret_key :- Secret key of AWS account where the bucket is present.
    \ncompress :- Set to true if we want to take\u00a0backup in the\u00a0compressed format.
    \nbase_path :- Name of the folder inside the s3 bucket where the backup is to be taken.<\/p>\n

    The second curl request would start taking the backup on AWS S3. Ensure the repository name is same (in our case,it is \u201cbackup\u201d). \u201csnapshot_prod\u201d is the name of the snapshot which would be stored on S3. \u2018wait_for_completion=true\u201d would not allow the command to release terminal until the backup is complete.<\/li>\n

  3. Once the data dump is complete, restore the data on Elasticsearch running on EC2 on AWS using \u201c_restore\u201d api provided by Elasticsearch.\n

    [js]curl -XPUT "http:\/\/:9200\/_snapshot\/myrepo" -d'{
    \n"type": "s3",
    \n"settings": {
    \n"bucket": "bucket-name",
    \n"region": "ap-southeast-1",
    \n"access_key": "——————————–",
    \n"secret_key": "—————————————————-",
    \n"compress": true,
    \n"base_path": "directory_name_inside_the_bucket"
    \n}
    \n}’;
    \ncurl -XPOST "http:\/\/:9200\/*\/_close";
    \ncurl -XPOST "http:\/\/:9200\/_snapshot\/myrepo\/snapshot_prod\/_restore?wait_for_completion=true";[\/js]<\/p>\n

    The first curl request would register the repository on the Elasticsearch cluster running on AWS EC2. We need to provide the same details under AWS s3 bucket setting which we have provided in step 2 as the data backup is on that s3 bucket.
    \nThe second curl request is to close all the existing indexes on Elasticsearch cluster running on AWS EC2. For our case, we have set up new Elasticsearch cluster on AWS EC2, we can optionally skip this step.
    \nThe third curl request is \u201cPOST\u201d request which would restore the Elasticsearch data from S3 to the Elasticsearch cluster running on AWS EC2.<\/p>\n

    Note<\/strong>:
    \n1. Ensure the repository name should be same which is \u201cmyrepo\u201d in our case.
    \n2. Use the same repository name \u201csnapshot_prod\u201d which we have used in step 2.<\/li>\n

  4. The completion step 3 will complete the Migration from Found to Elasticsearch hosted on AWS EC2.<\/li>\n<\/ol>\n

    This migration required a downtime depending upon the network and the size of the Elasticsearch data. I was able to migrate from Found to AWS EC2 Elasticsearch in forty minutes with a database size of 5Gb.<\/p>\n

    It is recommended that before migrating to AWS EC2 Elasticsearch Cluster:<\/strong><\/p>\n

      \n
    1. Ensure at least two nodes are up and running in the Elasticsearch cluster.<\/li>\n
    2. Use ec2 discovery provided by “cloud-aws” plugin. The plugin should be installed on all the nodes.<\/li>\n
    3. Disable swap space on EC2 in case it is enabled.<\/li>\n
    4. Set heap size equal to the half of RAM available on EC2.<\/li>\n
    5. Use \u201ci\u201d series EC2 instance as it is recommended by AWS for NoSql databases.<\/li>\n<\/ol>\n

      There are few things which need to be taken care after the migration is complete:<\/strong><\/p>\n

        \n
      1. Ensure regular (hourly or daily) snapshots are configured. Elasticsearch supports deltas of the database to be backed up. The step 2 curl request could be used to configure it after a\u00a0little modification.<\/li>\n
      2. Kopf is one plugin to take and restore the\u00a0backup.<\/li>\n
      3. Put alerts on rejected queue of various modules.<\/li>\n
      4. Use Shard filtering in case any non-customer facing application is fetching data from some specific indexes.<\/li>\n
      5. For monitoring, Marvel can be used but be careful as it daily creates one index. Ensure that it has been properly taken care off.<\/li>\n
      6. Implement Http-auth on Elasticsearch Cluster in case \u201chead\u201d plugin is being used.<\/li>\n<\/ol>\n

        Hope this blog helped you migrate Elasticsearch cluster from Found to AWS EC2. I will share more such use cases with you.<\/p>\n","protected":false},"excerpt":{"rendered":"

        Our DevOps team was using Found for one of our projects in the production environment. We have been facing a\u00a0problem with found where it\u2019s memory pressure frequently goes up and does not drop down so easily and until the time it remains up the Found was not able to serve\u00a0the requests. Then, we decided to […]<\/p>\n","protected":false},"author":154,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":7},"categories":[1174,2348,1],"tags":[2661,1547,4843,2548,3294,1524,3293,1252],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/34213"}],"collection":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/users\/154"}],"replies":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/comments?post=34213"}],"version-history":[{"count":1,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/34213\/revisions"}],"predecessor-version":[{"id":59862,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/34213\/revisions\/59862"}],"wp:attachment":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/media?parent=34213"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/categories?post=34213"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/tags?post=34213"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}