{"id":43442,"date":"2016-12-28T14:39:08","date_gmt":"2016-12-28T09:09:08","guid":{"rendered":"http:\/\/www.tothenew.com\/blog\/?p=43442"},"modified":"2016-12-28T15:28:30","modified_gmt":"2016-12-28T09:58:30","slug":"how-to-combine-aws-ephemeral-storage-by-using-raid-0","status":"publish","type":"post","link":"https:\/\/www.tothenew.com\/blog\/how-to-combine-aws-ephemeral-storage-by-using-raid-0\/","title":{"rendered":"How to Combine AWS Ephemeral Storage by Using RAID 0"},"content":{"rendered":"

As we know, ephemeral storage is SSD storage which is provided by AWS<\/a> freely with specific higher configuration of instances. It can be used to get the better disk I\/O, as this storage may lose data if instances stop. So we never use this ephemeral storage for important data.<\/p>\n

Below are the use cases where this combination can be used.<\/p>\n

    \n
  1. While uploading static content, we generally, we use CDN to serve the static content. First the Static content needs to be uploaded to the server, then it has to\u00a0be synced to S3. \u00a0Here we can use ephemeral storage as upload location.<\/li>\n
  2. Can also be used only to store\u00a0temporary data or SWAP (virtual memory) files.<\/li>\n
  3. Can be used for application logs, cache.<\/li>\n
  4. In case you need to format with different file-system types (ext3, ext4) to test something.<\/li>\n<\/ol>\n

    Some of the instances have two volumes of ephemeral storage. We can combine them with RAID technology (RAID 0 – striped volume). Refer below steps for the same. This is scripted as it\u00a0would be better to use in case we have to use both volumes of ephemeral storage as a single disk.<\/p>\n

    1.\u00a0Create File system (<\/strong>xvdb & <\/strong>xvdc):\u00a0In this blog, we have two Volumes attached with an instance. (each 1 GB)<\/strong><\/p>\n

    1.1. Check the volumes which are available (<\/strong>xvdb & <\/strong>xvdc)<\/strong><\/p>\n

    [root@ip-172-31-35-168 ~]# lsblk\r\n NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT\r\n xvda 202:0 0 8G 0 disk\r\n \u2514\u2500xvda1 202:1 0 8G 0 part \/\r\n xvdb 202:16 0 1G 0 disk <\/strong>\r\n xvdc 202:32 0 1G 0 disk<\/strong><\/pre>\n
    1.2. \u00a0As per above output, there is no file system for\u00a0xvdb and\u00a0xvdc, \u00a0First create files system here for\u00a0xvdb<\/strong>\"Combine<\/p>\n
    1.3. Do above steps for \u00a0xvdc<\/strong><\/p>\n
    1.4. Now we can see file system that has been created for both Volumes (xvdb1 & xvdc1)<\/strong><\/p>\n
    [root@ip-172-31-35-168 ~]# lsblk\r\n NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT\r\n xvda 202:0 0 8G 0 disk\r\n \u2514\u2500xvda1 202:1 0 8G 0 part \/\r\n xvdb 202:16 0 1G 0 disk\r\n \u2514\u2500xvdb1 202:17 0 1023M 0 part<\/strong>\r\n xvdc 202:32 0 1G 0 disk\r\n \u2514\u2500xvdc1 202:33 0 1023M 0 part <\/strong><\/pre>\n
    2. Create RAID 0 (striped): – Combine both of them as a single Volume.<\/strong><\/p>\n
    2.1. \u00a0Install the mdadm packages<\/strong><\/p>\n
    \/\/ yum install mdadm\r\n[root@ip-172-31-35-168 ~]# mdadm --create --verbose --auto=yes \/dev\/md0 --level=0 --raid-devices=2 \/dev\/xvdb1 \/dev\/xvdc1\r\n mdadm: chunk size defaults to 512K\r\n mdadm: Defaulting to version 1.2 metadata\r\n mdadm: array \/dev\/md0 started.<\/pre>\n
    2.2. Check the details of Volumes<\/strong><\/p>\n
    [root@ip-172-31-35-168 ~]# mdadm --details \/dev\/md0\r\nmdadm: unrecognized option '--details'\r\nUsage: mdadm --help\r\n for help\r\n[root@ip-172-31-35-168 ~]# mdadm --detail \/dev\/md0\r\n\/dev\/md0:\r\n Version : 1.2\r\n Creation Time : Mon Dec 19 06:15:53 2016\r\n Raid Level : raid0\r\n Array Size : 2094080 (2045.34 MiB 2144.34 MB)\r\n Raid Devices : 2\r\n Total Devices : 2\r\n Persistence : Superblock is persistent\r\n\r\n Update Time : Mon Dec 19 06:15:53 2016\r\n State : clean \r\n Active Devices : 2\r\nWorking Devices : 2\r\n Failed Devices : 0\r\n Spare Devices : 0\r\n\r\n Chunk Size : 512K\r\n\r\n Name : ip-172-31-35-168:0 (local to host ip-172-31-35-168)\r\n UUID : 446d819b:7bedab09:68580914:5d433e75\r\n Events : 0\r\n\r\n Number Major Minor RaidDevice State\r\n 0 202 17 0 active sync \/dev\/sdb1\r\n 1 202 33 1 active sync \/dev\/sdc1<\/pre>\n
    2.3. Verify again – We can see each filesystem belongs to md0 and showing 2 GB of space<\/strong><\/p>\n
    [root@ip-172-31-35-168 ~]# lsblk\r\n NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT\r\n xvda 202:0 0 8G 0 disk\r\n \u2514\u2500xvda1 202:1 0 8G 0 part \/\r\n xvdb 202:16 0 1G 0 disk\r\n \u2514\u2500xvdb1 202:17 0 1023M 0 part\r\n \u2514\u2500md0 9:0 0 2G 0 raid0<\/strong>\r\n xvdc 202:32 0 1G 0 disk\r\n \u2514\u2500xvdc1 202:33 0 1023M 0 part\r\n \u2514\u2500md0 9:0 0 2G 0 raid0<\/strong><\/pre>\n
    3. Format and mount the filesystem (\/dev\/md0)<\/strong><\/p>\n
    \/\/ To Format\r\n[root@ip-172-31-35-168 ~]# mkfs.ext4 \u00a0\/dev\/md0\r\n\r\n\/\/ To Mount temporary\r\n[root@ip-172-31-35-168 ~]# mount \/dev\/md0 \/mnt\r\n\r\n\/\/ To Mount Permanently\r\necho \/dev\/md0 \u00a0 \u00a0\/mnt \u00a0 \u00a0 \u00a0 \u00a0ext4 \u00a0 \u00a0defaults \u00a0 \u00a0 \u00a0 \u00a00 \u00a0 0 >> \/etc\/fstab<\/pre>\n
    \u00a04.\u00a0To automate above steps\u00a0i<\/i>n case of autoscaling<\/strong><\/p>\n
    \u00a04.1. Script to create partition and RAID.<\/strong><\/p>\n
    $ vim create_ephemeral_raid_md0.sh\r\n#!\/bin\/sh\r\n# Script to create software RAID0\r\n\r\nDISK1=\"\/dev\/xvdb\"\r\nDISK2=\"\/dev\/xvdc\"\r\n\r\n\/sbin\/fdisk $DISK1 < \/root\/admin_scripts\/partition_create_options\r\n\r\n\/sbin\/fdisk $DISK2 < \/root\/admin_scripts\/partition_create_options\r\n\r\n\/sbin\/mdadm --create --verbose --auto=yes \/dev\/md0 --level=0 --raid-devices=2 \/dev\/xvdb1 \/dev\/xvdc1\r\n\r\n\/sbin\/mkfs.ext4  \/dev\/md0<\/pre>\n
    \u00a0“partition_create_options” is a file that contains all options which are needed to create a file system.<\/p>\n
    \u00a04.2. Make an entry to auto start this script at boot<\/strong><\/p>\n
    \/\/ Add a cron at system boot @reboot \/bin\/sh <path of script>\/create_ephemeral_raid_md0.sh\r\n\r\n<\/pre>\n
    I hope the below blog helps the AWS DevOps engineers<\/a>\u00a0to use ephemeral storage.\u00a0 I will explain\u00a0the use cases in a\u00a0more descriptive way in my next blog.<\/p>\n","protected":false},"excerpt":{"rendered":"
    As we know, ephemeral storage is SSD storage which is provided by AWS freely with specific higher configuration of instances. It can be used to get the better disk I\/O, as this storage may lose data if instances stop. So we never use this ephemeral storage for important data. Below are the use cases where […]<\/p>\n","protected":false},"author":959,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":22},"categories":[1174,2348,1],"tags":[2366,1892,3236,4310,4311,4340,4309],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/43442"}],"collection":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/users\/959"}],"replies":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/comments?post=43442"}],"version-history":[{"count":0,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/43442\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/media?parent=43442"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/categories?post=43442"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/tags?post=43442"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}