Encrypting Data at rest if mandatory requirement compliance regulations such as PCI DSS and HIPAA. EBS, S3 has an option to encrypt the data stored in it. However, Instance Store volumes which provides temporary block level storage does have an option to encrypt the data stored in it. Customers need to use configure encryption using tools like dm-crypt.
Lets see how to automate encrypting the ephemeral volumes for Instance Store volumes on EC2 instances. I have written a user-data script which takes care of all the encryption setups.
The script will do the below to make it seamless for the customer:
- When you launch an instance or change the instance type, the script automatically detects the Ephemeral disks available to the instance type, setup encryption and make them available.
- Automatically installs the required packages for encryption if its not installed.
- The script also takes care of encryption in case of instance reboot or stop/start.
- The user-data script automatically take care of Stop/Start, Reboots and instance re-size.
- After stop/start you will lose all the data and configuration on the ephemeral volumes. This is expected, the user-data script will automatically detect this and setup encryption on the new disks and mounts it.
- When the instance is rebooted, the data persists. On a reboot we just need to decrypt the filesystem and mount it.
- The user data script will encrypt the volume and create a filesystem if its not already encrypted (These actions will be taken if you launch a fresh instance with this user-data or when you do a stop & start of your instance).
Now, if the volumes are already encrypted (this will happen on an instance reboot), the script will simply initialize the encrypted volume and mount it on the respective directory.
- The script will encrypt all the ephemeral volumes, create EXT4 filesystem and mount them under /encrypted_X (X being the last letter of the device name of the NVMe device name).
You may change the filesystem type, mount point as per your requirement, but you will need to update the logic to map volumes to correct mount point on the script.
- The encryption key/passphrase used is encrypted using AWS KMS and kept in a file in your S3 bucket. This is a one time setup and you could use the same encrypted passphrase file for your future installations. Instance need to be attached with an IAM instance profile with permission to download files from the S3 bucket.
Here are the step by step instructions:
1. Create an S3 Bucket for storing encrypted password file (or use an existing bucket in your account).
2. Create a IAM policy with the below policy document, this policy is going to be used for instance profile to allow it download the password file from s3 bucket:
Sign in to the AWS Management Console and navigate to the IAM console: https://console.aws.amazon.com/iam/home?region=us-east-1#/home
In the navigation pane, choose Policies, choose Create Policy, select Create Your Own Policy, name and describe the policy, and paste the following policy. Choose Create Policy.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1478729875000",
"Effect": "Allow",
"Action": [
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::/LuksInternalStorageKey"
]
}
]
}
---> Replace with the one you created/selected in step 1.
3. Create an EC2 role with the above policy.
In the IAM console, choose Roles, and then choose Create New Role.
In Step 1: Role Name, type your role name, and choose Next Step.
In Step 2: Select Role Type, choose Amazon EC2 and choose Next Step.
In Step 3: Established Trust, choose Next Step.
In Step 4: Attach Policy, choose the policy you created in Step 1
4. Create a KMS Key and add usage permission for the role you created above from the KMS console: https://console.aws.amazon.com/iam/home?region=us-east-1#/encryptionKeys/us-east-1
Step 1: Click ‘Create key’.
Step 2: Provide an Alias and Description.
Step 3: Add IAM role you created above as Key user under - ‘Define Key Usage Permissions’.
5. Create a secret password, encrypt it with KMS and store it in S3.
# aws --region us-east-1 kms encrypt --key-id 'alias/EncFSForEC2InternalStorageKey' --plaintext "ThisIs-a-SecretPassword" --query CiphertextBlob --output text | base64 --decode > LuksInternalStorageKey
--> Replace 'alias/EncFSForEC2InternalStorageKey' with the id of the KMS key you created above. The IAM user as which you run the above command should have permission to access the KMS key as well.
—> You should also replace the string “ThisIs-a-SecretPassword" with some strong passphrase.
6. Upload the encrypted key file to S3.
# aws s3 cp LuksInternalStorageKey s3:///LuksInternalStorageKey
7. Add the below userdata script to your instance at launch time. You can add/modify the user-data on a stopped instance as well.
I3, F1 instance type provides NVMe based instance store volumes and those disks are detected as /dev/nvmeXn1, X being the NVMe device number. Other instance types provides HDD/SSD based instance store volumes which gets detected as /dev/sdX. The user-data script detect the device names from EC2 meta-data in case of HDD/SSD and using nvme tool in case of NVMe based disks. I have written separate user data script for these volume types.
Use - user-data_NVMe_InstanceStore.txt for I3, F1 instances with NVMe disks.
Use - user-data_InstanceStore.txt for other instance types.
8. Once the instance is launched, you need to update the cloud-init configuration to make sure the user data runs every time the instance boots.
On Amazon Linux AMI based instances:
# sed -i 's/scripts-user/[scripts-user, always]/' /etc/cloud/cloud.cfg.d/00_defaults.cfg
On RHEL/CentOS instances:
# sed -i 's/scripts-user/[scripts-user, always]/' /etc/cloud/cloud.cfg
9. On Amazon Linux AMI instances, you might need disable crypt module on boot time to make sure the encrypted filesystems are not detected on the early boot stages:
# sed -i 's/#omit_dracutmodules+=""/omit_dracutmodules+="crypt"/' /etc/dracut.conf
# dracut --force
User Data Scripts:
-- user-data_InstanceStore.txt
Regards,
Jay
Lets see how to automate encrypting the ephemeral volumes for Instance Store volumes on EC2 instances. I have written a user-data script which takes care of all the encryption setups.
The script will do the below to make it seamless for the customer:
- When you launch an instance or change the instance type, the script automatically detects the Ephemeral disks available to the instance type, setup encryption and make them available.
- Automatically installs the required packages for encryption if its not installed.
- The script also takes care of encryption in case of instance reboot or stop/start.
- The user-data script automatically take care of Stop/Start, Reboots and instance re-size.
- After stop/start you will lose all the data and configuration on the ephemeral volumes. This is expected, the user-data script will automatically detect this and setup encryption on the new disks and mounts it.
- When the instance is rebooted, the data persists. On a reboot we just need to decrypt the filesystem and mount it.
- The user data script will encrypt the volume and create a filesystem if its not already encrypted (These actions will be taken if you launch a fresh instance with this user-data or when you do a stop & start of your instance).
Now, if the volumes are already encrypted (this will happen on an instance reboot), the script will simply initialize the encrypted volume and mount it on the respective directory.
- The script will encrypt all the ephemeral volumes, create EXT4 filesystem and mount them under /encrypted_X (X being the last letter of the device name of the NVMe device name).
You may change the filesystem type, mount point as per your requirement, but you will need to update the logic to map volumes to correct mount point on the script.
- The encryption key/passphrase used is encrypted using AWS KMS and kept in a file in your S3 bucket. This is a one time setup and you could use the same encrypted passphrase file for your future installations. Instance need to be attached with an IAM instance profile with permission to download files from the S3 bucket.
Here are the step by step instructions:
1. Create an S3 Bucket for storing encrypted password file (or use an existing bucket in your account).
2. Create a IAM policy with the below policy document, this policy is going to be used for instance profile to allow it download the password file from s3 bucket:
Sign in to the AWS Management Console and navigate to the IAM console: https://console.aws.amazon.com/iam/home?region=us-east-1#/home
In the navigation pane, choose Policies, choose Create Policy, select Create Your Own Policy, name and describe the policy, and paste the following policy. Choose Create Policy.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1478729875000",
"Effect": "Allow",
"Action": [
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::
]
}
]
}
---> Replace
3. Create an EC2 role with the above policy.
In the IAM console, choose Roles, and then choose Create New Role.
In Step 1: Role Name, type your role name, and choose Next Step.
In Step 2: Select Role Type, choose Amazon EC2 and choose Next Step.
In Step 3: Established Trust, choose Next Step.
In Step 4: Attach Policy, choose the policy you created in Step 1
4. Create a KMS Key and add usage permission for the role you created above from the KMS console: https://console.aws.amazon.com/iam/home?region=us-east-1#/encryptionKeys/us-east-1
Step 1: Click ‘Create key’.
Step 2: Provide an Alias and Description.
Step 3: Add IAM role you created above as Key user under - ‘Define Key Usage Permissions’.
5. Create a secret password, encrypt it with KMS and store it in S3.
# aws --region us-east-1 kms encrypt --key-id 'alias/EncFSForEC2InternalStorageKey' --plaintext "ThisIs-a-SecretPassword" --query CiphertextBlob --output text | base64 --decode > LuksInternalStorageKey
--> Replace 'alias/EncFSForEC2InternalStorageKey' with the id of the KMS key you created above. The IAM user as which you run the above command should have permission to access the KMS key as well.
—> You should also replace the string “ThisIs-a-SecretPassword" with some strong passphrase.
6. Upload the encrypted key file to S3.
# aws s3 cp LuksInternalStorageKey s3://
7. Add the below userdata script to your instance at launch time. You can add/modify the user-data on a stopped instance as well.
I3, F1 instance type provides NVMe based instance store volumes and those disks are detected as /dev/nvmeXn1, X being the NVMe device number. Other instance types provides HDD/SSD based instance store volumes which gets detected as /dev/sdX. The user-data script detect the device names from EC2 meta-data in case of HDD/SSD and using nvme tool in case of NVMe based disks. I have written separate user data script for these volume types.
Use - user-data_NVMe_InstanceStore.txt for I3, F1 instances with NVMe disks.
Use - user-data_InstanceStore.txt for other instance types.
8. Once the instance is launched, you need to update the cloud-init configuration to make sure the user data runs every time the instance boots.
On Amazon Linux AMI based instances:
# sed -i 's/scripts-user/[scripts-user, always]/' /etc/cloud/cloud.cfg.d/00_defaults.cfg
On RHEL/CentOS instances:
# sed -i 's/scripts-user/[scripts-user, always]/' /etc/cloud/cloud.cfg
9. On Amazon Linux AMI instances, you might need disable crypt module on boot time to make sure the encrypted filesystems are not detected on the early boot stages:
# sed -i 's/#omit_dracutmodules+=""/omit_dracutmodules+="crypt"/' /etc/dracut.conf
# dracut --force
User Data Scripts:
-- user-data_InstanceStore.txt
#!/bin/bash-- user-data_NVMe_InstanceStore.txt
REGION=
S3_Bucket=
# Install required packages if not installed
[ $(which unzip) ] || yum -y install unzip
[ $(which aws) ] || "$(/usr/bin/curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip" ; /usr/bin/unzip -o awscli-bundle.zip; ./awscli-bundle/install -b /usr/bin/aws ; rm -rf /awscli-bundle*)"
[ $(which cryptsetup) ] || yum install -y cryptsetup
# Get the encrypted password file from s3
/usr/bin/aws s3 cp s3://${S3_Bucket}/LuksInternalStorageKey .
# Decrypt and store the passphrase in a variable
LuksClearTextKey=$(/usr/bin/aws --region {REGION} kms decrypt --ciphertext-blob fileb://LuksInternalStorageKey --output text --query Plaintext | base64 --decode)
for ephemeral in $(curl -s http://169.254.169.254/latest/meta-data/block-device-mapping/ |grep ephemeral)
do
DEV=$(curl -s http://169.254.169.254/latest/meta-data/block-device-mapping/$ephemeral |sed 's/s/\/dev\/xv/g')
DEV_1=`echo "${DEV: -1}"`
[ "$(/bin/mount | grep -i ${DEV})" ] && /bin/umount ${DEV}
TYPE=`/usr/bin/file -sL ${DEV} | awk '{print $2}'`
if [ $TYPE == "LUKS" ]
then
# Open and initialize the encryped volume
/bin/echo "$LuksClearTextKey" | /sbin/cryptsetup luksOpen ${DEV} encfs_${DEV_1}
# Check and create mount point if not exists
[ -d /encrypted_${DEV_1} ] || /bin/mkdir /encrypted_${DEV_1}
# Mount the filsystem
/bin/mount /dev/mapper/encfs_${DEV_1} /encrypted_${DEV_1}
else
# Encrypt the volume sub
/bin/echo "$LuksClearTextKey" | cryptsetup -y luksFormat ${DEV}
# Open and initialize the encryped volume
/bin/echo "$LuksClearTextKey" | cryptsetup luksOpen ${DEV} encfs_${DEV_1}
# create a filesystem on the encrypted volume, mount it on the required directory
/sbin/mkfs.ext4 /dev/mapper/encfs_${DEV_1}
[ -d /encrypted_${DEV_1} ] || /bin/mkdir /encrypted_${DEV_1}
/bin/mount /dev/mapper/encfs_${DEV_1} /encrypted_${DEV_1}
fi
done
# Unset the passphrase variable and remove the encrypted password file.
unset LuksInternalStorageKey
rm LuksInternalStorageKey
#!/bin/bash
#set -x
REGION=
S3_Bucket=
# Install required packages if not installed
[ $(which unzip) ] || yum install -y zip unzip
[ $(which python) ] || yum install -y python
[ $(which aws) ] || "$(/usr/bin/curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip" ; /usr/bin/unzip -o awscli-bundle.zip; ./awscli-bundle/install -b /usr/bin/aws ; rm -rf /awscli-bundle*)"
[ "$(which cryptsetup)" ] || yum install -y cryptsetup
[ "$(which nvme)" ] || yum install -y nvme-cli
# Get the encrypted password file from s3
/usr/bin/aws s3 cp s3://${S3_Bucket}/LuksInternalStorageKey .
# Decrypt and store the passphrase in a variable
LuksClearTextKey=$(/usr/bin/aws --region ${REGION} kms decrypt --ciphertext-blob fileb://LuksInternalStorageKey --output text --query Plaintext | base64 --decode)
for ephemeral in $(nvme list | grep dev | awk {'print $1'})
do
DEV_1=$(echo "${ephemeral:9:1}")
[ "$(/bin/mount | grep -i ${ephemeral})" ] && /bin/umount ${ephemeral}
TYPE=`/usr/bin/file -sL ${ephemeral} | awk '{print $2}'`
if [ $TYPE == "LUKS" ]
then
# Open and initialize the encryped volume
/bin/echo "$LuksClearTextKey" | /sbin/cryptsetup luksOpen ${ephemeral} encfs_${DEV_1}
# Check and create mount point if not exists
[ -d /encrypted_${DEV_1} ] || /bin/mkdir /encrypted_${DEV_1}
# Mount the filsystem
/bin/mount /dev/mapper/encfs_${DEV_1} /encrypted_${DEV_1}
else
# Encrypt the volume sub
/bin/echo "$LuksClearTextKey" | cryptsetup -y luksFormat ${ephemeral}
# Open and initialize the encryped volume
/bin/echo "$LuksClearTextKey" | cryptsetup luksOpen ${ephemeral} encfs_${DEV_1}
# create a filesystem on the encrypted volume, mount it on the required directory
/sbin/mkfs.ext4 /dev/mapper/encfs_${DEV_1}
[ -d /encrypted_${DEV_1} ] || /bin/mkdir /encrypted_${DEV_1}
/bin/mount /dev/mapper/encfs_${DEV_1} /encrypted_${DEV_1}
fi
done
# Unset the passphrase variable and remove the encrypted password file.
unset LuksInternalStorageKey
rm LuksInternalStorageKey
Regards,
Jay
