EFS

EFS - Elastic Filesystem

Other methods besides DD

How can I copy data to and from Amazon EFS in parallel to maximize performance on my EC2 instance?

https://aws.amazon.com/premiumsupport/knowledge-center/efs-copy-data-in-parallel/

DD is not parallel, so there are other methods to create files much quicker than DD. Below is a process on how you can do so:

1) Create ten thousand small files in your local storage on your instance.

$ mkdir /tmp/efs; for each in $(seq 1 10000); do SUFFIX=$(mktemp -u EFS-XXXXXX); sudo dd if=/dev/zero of=/tmp/efs/${SUFFIX} bs=64k count=1; done

2) To test copy the files from your instance to the EFS file system:

Serial method: 
$ cd /tmp/efs; time sudo find -L . -maxdepth 1 -type f -exec rsync -aR '{}' /efs/ \;

Parallel method: 
$ cd /tmp/efs; time find -L . -maxdepth 1 -type f | sudo parallel rsync -aR {} /efs/

3) To remove:

Serial method: 
$ cd /mnt/efs; time sudo find -L . -maxdepth 1 -exec rm -rfv {} \;

Parallel method: 
$ cd /mnt/efs; find -L . -maxdepth 1 -type f | sudo parallel rm -rfv {}

TESTED BY MOUNTING EFS TO AN INSTANCE, AND CREATED DUMMY FILES IN THE LOCAL STORAGE. THEN USE GNU PARALLEL AND RSYNC TO COPY 20GB of files to my EFS

https://aws.amazon.com/premiumsupport/knowledge-center/efs-copy-data-in-parallel/

1) Create EFS filesystem

Modify/adjust security group of mount targets to allow inbound connection from EC2 instance.
Creating security groups using the AWS Management Console: https://docs.aws.amazon.com/efs/latest/ug/accessing-fs-create-security-groups.html#create-security-groups-console
Ensure the security group of the EC2 instance allows outbound connection to the EFS filesystem (or open to ALL)

2) Connect to your EC2 instance and ensure the amazon-efs-utils package is installed. This package contains the EFS mount helper. The Amazon EFS mount helper simplifies mounting your file systems.

$ sudo yum install -y amazon-efs-utils (as you are using Amazon Linux, this should already be installed)

3) Make a directory to mount the EFS filesystem:

$ mkdir ~/efs-mount-point

4) Next mount the EFS filesystem using the EFS mount helper:

https://docs.aws.amazon.com/efs/latest/ug/mounting-fs-mount-helper.html#mounting-fs-mount-helper-ec2

$  sudo mount -t efs fs-b32311b6:/ ~/efs-mount-point/

That should complete the mounting process. Next we can increase the EFS filesystem size by running the following commands: https://aws.amazon.com/premiumsupport/knowledge-center/efs-copy-data-in-parallel/

5) The command below will create a temporary directory in your local storage. It will then create 20 x 1GB files within that directory.

$ mkdir /tmp/efs;  for each in $(seq 1 20); do SUFFIX=$(mktemp -u EFS-XXXXXX); sudo dd if=/dev/zero of=/tmp/efs/${SUFFIX} bs=1M count=1024; done

To confirm the number of files created, you can run the following command:

$ ls -alh /tmp/efs > files_created.txt ; grep -i EFS "files_created.txt" | wc -l

6) Next, run the following commands:

$ sudo amazon-linux-extras install epel
$ sudo yum install nload sysstat parallel -y
$ sudo yum install time -y
$ sudo yum install rsync -y

7) Once you have created the files and installed GNU Parallel, run the following command to copy the files from your instance to the EFS file system:

# didn't work:
$ cd /tmp/efs; time find -L . -maxdepth 1 -type f | sudo parallel rsync -aR {} ~/efs-mount-point

# worked
$ cd /tmp/efs; sudo time find -L . -type f | parallel rsync -avR {} ~/efs-mount-point

Example output while its running:

sent 1,074,004,098 bytes  received 35 bytes  49,953,680.60 bytes/sec
total size is 1,073,741,824  speedup is 1.00
sending incremental file list
EFS-UwWFli

sent 1,074,004,099 bytes  received 35 bytes  49,953,680.65 bytes/sec
total size is 1,073,741,824  speedup is 1.00
sending incremental file list
EFS-yqR525

sent 1,074,004,099 bytes  received 35 bytes  49,953,680.65 bytes/sec
total size is 1,073,741,824  speedup is 1.00
sending incremental file list
EFS-MJjQfR

Once it's completed:

sent 1,074,004,099 bytes  received 35 bytes  52,390,445.56 bytes/sec
total size is 1,073,741,824  speedup is 1.00

I was able to confirm my EFS filesystem grew from 6KiB to 20GiB.