Issue: How to do faster copy and delete operations on EFS file systems.
Environment:
Amazon Linux
Ubuntu Server
Amazon EFS
Solution:
To optimize copy and delete operations on EFS file systems, you can use the GNU Parallel shell tool for executing jobs in parallel. By doing this you will be able to complete these tasks faster than using the normal serial method.
1.a. Install the NFS utilities and the GNU parallel package on Amazon Linux.
[ec2-user ~]$ sudo yum install nfs-utils -y
[ec2-user ~]$ sudo yum install parallel
1.b. Install the NFS utilities and the GNU parallel package on Ubuntu Server.
[ubuntu ~]$ sudo apt-get install nfs-common parallel -y
1.c. Install from source:
[ec2-user ~]$ cd /tmp; wget http://ftp.gnu.org/gnu/parallel/parallel-latest.tar.bz2
[ec2-user ~]$ tar -xvf parallel-20120122.tar.bz2; cd parallel-20170822
[ec2-user ~]$ sudo yum groupinstall 'development tools' -y
[ec2-user ~]$ make; ./configure && sudo make install
2. Create a temporary directory and mount the EFS filesystem.
[ec2-user ~]$ sudo mkdir /mnt/efs; sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 fs-XXXXXXXX.efs.REGION.amazonaws.com:/ /mnt/efs
3. Create ten thousand small files local in your instance.
[ec2-user ~]$ mkdir /tmp/efs; for each in $(seq 1 10000); do SUFFIX=$(mktemp -u _XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX); sudo dd if=/dev/zero of=/tmp/efs/${SUFFIX} bs=64k count=1; done
4. Copy the files from your instance to the EFS file system using the parallel method.
[ec2-user ~]$ cd /tmp/efs; find -L . -maxdepth 1 -type f | sudo parallel rsync -avR {} /mnt/efs/
5. Delete the files from the EFS file system using the parallel method.
[ec2-user ~]$ cd /mnt/efs; find -L . -maxdepth 1 -type f | sudo parallel rm -rfv {}
Test:
The following output is from my tests, using an EFS file system (General Purpose) mounted on a t2.micro instance type:
1. Copy ten thousand files from my EC2 instance to my EFS file system, using the normal serial method.
[ec2-user ~]$ cd /tmp/efs; time sudo find -L . -maxdepth 1 -type f -exec rsync -avR '{}' /mnt/efs/ \;
real 20m8.947s
user 0m0.060s
sys 0m0.980s
2. Copy ten thousand files from my EC2 instance to my EFS file system, using the parallel method.
[ec2-user ~]$ cd /tmp/efs; time find -L . -maxdepth 1 -type f | sudo parallel rsync -avR {} /mnt/efs/
real 5m34.264s
user 0m8.308s
sys 0m6.904s
3. Delete ten thousand files from my EFS file system, using the normal serial method.
[ec2-user ~]$ cd /mnt/efs; time sudo find -L . -maxdepth 1 -exec rm -rfv {} \;
real 2m24.799s
user 0m0.124s
sys 0m1.240s
4. Delete ten thousand files from my EFS file system, using the parallel method.
[ec2-user ~]$ cd /mnt/efs; find -L . -maxdepth 1 -type f | sudo parallel rm -rfv {}
real 1m55.153s
user 0m7.988s
sys 0m6.972s
Recursively copy :
To add to this amazing article as most us are using this as a sample for our customers I would like to mention that the examples above will copy all files from SRC only, not recursively into /SRC. If you need recursively copy you will have two option with rsync:
Loop:
find /SRC/ -type d | while read -r c; do cd "$c"; find -L . -maxdepth 1 -type f | parallel rsync --avR {} /DST; done
As the above allows you the possibility to run parallel copy and recursively copy the data from SRC/ to DST it also introduces a performance penalty as the loop have to read to each folder recursively and fire parallel copies only of the content of that location.
List Creation:
Create List
rsync -avR --dry-run /SRC /DST > lsit.log
Run the command:
cat list.log | parallel --will-cite -j 100 rsync -avR {} /DST/
The above is a much simpler approach, what basically do is to create a list of all files/folders recursively on /SRC and fire 100 parallel copies reading the path of the files to copy from the list. This allows the copies to be much more efficient as having less over the head.
[2] http://www.gnu.org/software/parallel/man.html#EXAMPLE:-Parallelizing-rsync
In the realm of modern computing, where data storage and retrieval speed are paramount, understanding the performance of storage solutions is crucial. One of the fundamental components of this landscape is Linux block volume performance calculation. Whether you're a system administrator, a developer, or an enthusiast, delving into the intricacies of block volume performance, including Fio-based tests, can empower you to make informed decisions about storage setups. In this blog post, we'll demystify the concepts behind Linux block volume performance calculation and explore the key factors that influence it, along with practical Fio-based tests.
Understanding Block Volumes:
Block volumes are a type of storage solution commonly used in modern IT infrastructures. They provide raw storage space that can be partitioned and formatted according to the user's needs. These volumes are often found in virtual machines, cloud instances, and even physical servers. They are characterized by their ability to handle data at the block level, meaning data is read from and written to storage in fixed-size blocks.
Factors Influencing Block Volume Performance:
Several factors play a pivotal role in determining the performance of Linux block volumes. Understanding these factors helps optimize storage systems for better efficiency and responsiveness.
1. I/O Operations Per Second (IOPS): IOPS refers to the number of input/output operations a storage device can handle in a second. It is a key metric in assessing storage responsiveness. The higher the IOPS, the faster the storage system can read from or write to the block volume.
2. Throughput: Throughput measures the amount of data that can be transferred between the storage device and the system in a given period. It's usually measured in megabytes or gigabytes per second. Throughput is a crucial metric when dealing with large data transfers.
3. Latency: Latency is the delay between initiating a data request and receiving the first byte of data. Lower latency indicates a more responsive storage system. Excessive latency can lead to delays in data-intensive operations.
4. Queue Depth: Queue depth refers to the number of I/O requests that can be in the queue to the storage device at a given time. A higher queue depth can lead to improved performance, especially in scenarios with concurrent I/O operations.
Calculating Block Volume Performance:
While calculating precise block volume performance can be intricate, here's a simplified approach:
1. IOPS Calculation: Determine the total IOPS required by considering the application's read and write demands. Divide this total by the number of block volumes to distribute the load. It's important to consider peak I/O requirements.
2. Throughput Calculation: Calculate the required throughput by estimating the data transfer needs of the application. Divide this by the number of block volumes for load distribution.
3. Latency Estimation: Latency is affected by various factors, including the speed of the storage media and the efficiency of the underlying technology. Faster media and optimized configurations lead to lower latency.
[opc@jay ~]$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 46.6G 0 disk
├─sda1 8:1 0 512M 0 part /boot/efi
├─sda2 8:2 0 8G 0 part [SWAP]
└─sda3 8:3 0 38.1G 0 part /
sdb 8:16 0 47G 0 disk >>>>> the disk is detected as /dev/sdb
├─sdb1 8:17 0 512M 0 part >>>>> Boot partition
├─sdb2 8:18 0 8G 0 part
└─sdb3 8:19 0 38.1G 0 part >>>>> Root partition
[opc@jay ~]$# mkdir /recovery
# mount /dev/sdb3 /recovery
You might need to specify -o nouuid for some OS.
# Further relax access to the default document root:You could remove those tabbed comments as well using "sed" or using "egrep" with proper Regex.
#
# Possible values for the Options directive are "None", "All",
# or any combination of:
# Indexes Includes FollowSymLinks SymLinksifOwnerMatch ExecCGI MultiViews
#!/bin/bash-- user-data_NVMe_InstanceStore.txt
REGION=
S3_Bucket=
# Install required packages if not installed
[ $(which unzip) ] || yum -y install unzip
[ $(which aws) ] || "$(/usr/bin/curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip" ; /usr/bin/unzip -o awscli-bundle.zip; ./awscli-bundle/install -b /usr/bin/aws ; rm -rf /awscli-bundle*)"
[ $(which cryptsetup) ] || yum install -y cryptsetup
# Get the encrypted password file from s3
/usr/bin/aws s3 cp s3://${S3_Bucket}/LuksInternalStorageKey .
# Decrypt and store the passphrase in a variable
LuksClearTextKey=$(/usr/bin/aws --region {REGION} kms decrypt --ciphertext-blob fileb://LuksInternalStorageKey --output text --query Plaintext | base64 --decode)
for ephemeral in $(curl -s http://169.254.169.254/latest/meta-data/block-device-mapping/ |grep ephemeral)
do
DEV=$(curl -s http://169.254.169.254/latest/meta-data/block-device-mapping/$ephemeral |sed 's/s/\/dev\/xv/g')
DEV_1=`echo "${DEV: -1}"`
[ "$(/bin/mount | grep -i ${DEV})" ] && /bin/umount ${DEV}
TYPE=`/usr/bin/file -sL ${DEV} | awk '{print $2}'`
if [ $TYPE == "LUKS" ]
then
# Open and initialize the encryped volume
/bin/echo "$LuksClearTextKey" | /sbin/cryptsetup luksOpen ${DEV} encfs_${DEV_1}
# Check and create mount point if not exists
[ -d /encrypted_${DEV_1} ] || /bin/mkdir /encrypted_${DEV_1}
# Mount the filsystem
/bin/mount /dev/mapper/encfs_${DEV_1} /encrypted_${DEV_1}
else
# Encrypt the volume sub
/bin/echo "$LuksClearTextKey" | cryptsetup -y luksFormat ${DEV}
# Open and initialize the encryped volume
/bin/echo "$LuksClearTextKey" | cryptsetup luksOpen ${DEV} encfs_${DEV_1}
# create a filesystem on the encrypted volume, mount it on the required directory
/sbin/mkfs.ext4 /dev/mapper/encfs_${DEV_1}
[ -d /encrypted_${DEV_1} ] || /bin/mkdir /encrypted_${DEV_1}
/bin/mount /dev/mapper/encfs_${DEV_1} /encrypted_${DEV_1}
fi
done
# Unset the passphrase variable and remove the encrypted password file.
unset LuksInternalStorageKey
rm LuksInternalStorageKey
#!/bin/bash
#set -x
REGION=
S3_Bucket=
# Install required packages if not installed
[ $(which unzip) ] || yum install -y zip unzip
[ $(which python) ] || yum install -y python
[ $(which aws) ] || "$(/usr/bin/curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip" ; /usr/bin/unzip -o awscli-bundle.zip; ./awscli-bundle/install -b /usr/bin/aws ; rm -rf /awscli-bundle*)"
[ "$(which cryptsetup)" ] || yum install -y cryptsetup
[ "$(which nvme)" ] || yum install -y nvme-cli
# Get the encrypted password file from s3
/usr/bin/aws s3 cp s3://${S3_Bucket}/LuksInternalStorageKey .
# Decrypt and store the passphrase in a variable
LuksClearTextKey=$(/usr/bin/aws --region ${REGION} kms decrypt --ciphertext-blob fileb://LuksInternalStorageKey --output text --query Plaintext | base64 --decode)
for ephemeral in $(nvme list | grep dev | awk {'print $1'})
do
DEV_1=$(echo "${ephemeral:9:1}")
[ "$(/bin/mount | grep -i ${ephemeral})" ] && /bin/umount ${ephemeral}
TYPE=`/usr/bin/file -sL ${ephemeral} | awk '{print $2}'`
if [ $TYPE == "LUKS" ]
then
# Open and initialize the encryped volume
/bin/echo "$LuksClearTextKey" | /sbin/cryptsetup luksOpen ${ephemeral} encfs_${DEV_1}
# Check and create mount point if not exists
[ -d /encrypted_${DEV_1} ] || /bin/mkdir /encrypted_${DEV_1}
# Mount the filsystem
/bin/mount /dev/mapper/encfs_${DEV_1} /encrypted_${DEV_1}
else
# Encrypt the volume sub
/bin/echo "$LuksClearTextKey" | cryptsetup -y luksFormat ${ephemeral}
# Open and initialize the encryped volume
/bin/echo "$LuksClearTextKey" | cryptsetup luksOpen ${ephemeral} encfs_${DEV_1}
# create a filesystem on the encrypted volume, mount it on the required directory
/sbin/mkfs.ext4 /dev/mapper/encfs_${DEV_1}
[ -d /encrypted_${DEV_1} ] || /bin/mkdir /encrypted_${DEV_1}
/bin/mount /dev/mapper/encfs_${DEV_1} /encrypted_${DEV_1}
fi
done
# Unset the passphrase variable and remove the encrypted password file.
unset LuksInternalStorageKey
rm LuksInternalStorageKey