As we know, ephemeral storage is SSD storage which is provided by AWS freely with specific higher configuration of instances. It can be used to get the better disk I/O, as this storage may lose data if instances stop. So we never use this ephemeral storage for important data.
Below are the use cases where this combination can be used.
- While uploading static content, we generally, we use CDN to serve the static content. First the Static content needs to be uploaded to the server, then it has to be synced to S3. Here we can use ephemeral storage as upload location.
- Can also be used only to store temporary data or SWAP (virtual memory) files.
- Can be used for application logs, cache.
- In case you need to format with different file-system types (ext3, ext4) to test something.
Some of the instances have two volumes of ephemeral storage. We can combine them with RAID technology (RAID 0 – striped volume). Refer below steps for the same. This is scripted as it would be better to use in case we have to use both volumes of ephemeral storage as a single disk.
1. Create File system (xvdb & xvdc): In this blog, we have two Volumes attached with an instance. (each 1 GB)
1.1. Check the volumes which are available (xvdb & xvdc)
[root@ip-172-31-35-168 ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT xvda 202:0 0 8G 0 disk └─xvda1 202:1 0 8G 0 part / xvdb 202:16 0 1G 0 disk xvdc 202:32 0 1G 0 disk
1.2. As per above output, there is no file system for xvdb and xvdc, First create files system here for xvdb
1.3. Do above steps for xvdc
1.4. Now we can see file system that has been created for both Volumes (xvdb1 & xvdc1)
[root@ip-172-31-35-168 ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT xvda 202:0 0 8G 0 disk └─xvda1 202:1 0 8G 0 part / xvdb 202:16 0 1G 0 disk └─xvdb1 202:17 0 1023M 0 part xvdc 202:32 0 1G 0 disk └─xvdc1 202:33 0 1023M 0 part
2. Create RAID 0 (striped): – Combine both of them as a single Volume.
2.1. Install the mdadm packages
// yum install mdadm [root@ip-172-31-35-168 ~]# mdadm --create --verbose --auto=yes /dev/md0 --level=0 --raid-devices=2 /dev/xvdb1 /dev/xvdc1 mdadm: chunk size defaults to 512K mdadm: Defaulting to version 1.2 metadata mdadm: array /dev/md0 started.
2.2. Check the details of Volumes
[root@ip-172-31-35-168 ~]# mdadm --details /dev/md0 mdadm: unrecognized option '--details' Usage: mdadm --help for help [root@ip-172-31-35-168 ~]# mdadm --detail /dev/md0 /dev/md0: Version : 1.2 Creation Time : Mon Dec 19 06:15:53 2016 Raid Level : raid0 Array Size : 2094080 (2045.34 MiB 2144.34 MB) Raid Devices : 2 Total Devices : 2 Persistence : Superblock is persistent Update Time : Mon Dec 19 06:15:53 2016 State : clean Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 Chunk Size : 512K Name : ip-172-31-35-168:0 (local to host ip-172-31-35-168) UUID : 446d819b:7bedab09:68580914:5d433e75 Events : 0 Number Major Minor RaidDevice State 0 202 17 0 active sync /dev/sdb1 1 202 33 1 active sync /dev/sdc1
2.3. Verify again – We can see each filesystem belongs to md0 and showing 2 GB of space
[root@ip-172-31-35-168 ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT xvda 202:0 0 8G 0 disk └─xvda1 202:1 0 8G 0 part / xvdb 202:16 0 1G 0 disk └─xvdb1 202:17 0 1023M 0 part └─md0 9:0 0 2G 0 raid0 xvdc 202:32 0 1G 0 disk └─xvdc1 202:33 0 1023M 0 part └─md0 9:0 0 2G 0 raid0
3. Format and mount the filesystem (/dev/md0)
// To Format [root@ip-172-31-35-168 ~]# mkfs.ext4 /dev/md0 // To Mount temporary [root@ip-172-31-35-168 ~]# mount /dev/md0 /mnt // To Mount Permanently echo /dev/md0 /mnt ext4 defaults 0 0 >> /etc/fstab
4. To automate above steps in case of autoscaling
4.1. Script to create partition and RAID.
$ vim create_ephemeral_raid_md0.sh #!/bin/sh # Script to create software RAID0 DISK1="/dev/xvdb" DISK2="/dev/xvdc" /sbin/fdisk $DISK1 < /root/admin_scripts/partition_create_options /sbin/fdisk $DISK2 < /root/admin_scripts/partition_create_options /sbin/mdadm --create --verbose --auto=yes /dev/md0 --level=0 --raid-devices=2 /dev/xvdb1 /dev/xvdc1 /sbin/mkfs.ext4 /dev/md0
“partition_create_options” is a file that contains all options which are needed to create a file system.
4.2. Make an entry to auto start this script at boot
// Add a cron at system boot @reboot /bin/sh <path of script>/create_ephemeral_raid_md0.sh
I hope the below blog helps the AWS DevOps engineers to use ephemeral storage. I will explain the use cases in a more descriptive way in my next blog.