Ardika Sulistija: Created page with "EFS - Elastic Filesystem ====Other methods besides DD==== ---- * How can I copy data to and from Amazon EFS in parallel to maximize performance on my EC2 instance?
https://aws.amazon.com/premiumsupport/knowledge-center/efs-copy-data-in-parallel/ DD is not parallel, so there are other methods to create files much quicker than DD. Below is a process on how you can do so: 1) Create ten thousand small files in your local storage on your instance. $ mkdir /t..."

2024-08-27T00:10:49Z

Created page with "EFS - Elastic Filesystem ====Other methods besides DD==== ---- * How can I copy data to and from Amazon EFS in parallel to maximize performance on my EC2 instance? https://aws.amazon.com/premiumsupport/knowledge-center/efs-copy-data-in-parallel/ DD is not parallel, so there are other methods to create files much quicker than DD. Below is a process on how you can do so: 1) Create ten thousand small files in your local storage on your instance. $ mkdir /t..."

New page

EFS - Elastic Filesystem

====Other methods besides DD====
----

* How can I copy data to and from Amazon EFS in parallel to maximize performance on my EC2 instance?
https://aws.amazon.com/premiumsupport/knowledge-center/efs-copy-data-in-parallel/

DD is not parallel, so there are other methods to create files much quicker than DD. Below is a process on how you can do so:

1) Create ten thousand small files in your local storage on your instance.

$ mkdir /tmp/efs; for each in $(seq 1 10000); do SUFFIX=$(mktemp -u EFS-XXXXXX); sudo dd if=/dev/zero of=/tmp/efs/${SUFFIX} bs=64k count=1; done

2) To test copy the files from your instance to the EFS file system:

Serial method:
$ cd /tmp/efs; time sudo find -L . -maxdepth 1 -type f -exec rsync -aR '{}' /efs/ \;

Parallel method:
$ cd /tmp/efs; time find -L . -maxdepth 1 -type f | sudo parallel rsync -aR {} /efs/

3) To remove:

Serial method:
$ cd /mnt/efs; time sudo find -L . -maxdepth 1 -exec rm -rfv {} \;

Parallel method:
$ cd /mnt/efs; find -L . -maxdepth 1 -type f | sudo parallel rm -rfv {}

====TESTED BY MOUNTING EFS TO AN INSTANCE, AND CREATED DUMMY FILES IN THE LOCAL STORAGE. THEN USE GNU PARALLEL AND RSYNC TO COPY 20GB of files to my EFS====
----

https://aws.amazon.com/premiumsupport/knowledge-center/efs-copy-data-in-parallel/

1) Create EFS filesystem
* Modify/adjust security group of mount targets to allow inbound connection from EC2 instance.
* Creating security groups using the AWS Management Console: https://docs.aws.amazon.com/efs/latest/ug/accessing-fs-create-security-groups.html#create-security-groups-console
* Ensure the security group of the EC2 instance allows outbound connection to the EFS filesystem (or open to ALL)

2) Connect to your EC2 instance and ensure the amazon-efs-utils package is installed. This package contains the EFS mount helper. The Amazon EFS mount helper simplifies mounting your file systems.
* https://docs.aws.amazon.com/efs/latest/ug/installing-amazon-efs-utils.html
* https://docs.aws.amazon.com/efs/latest/ug/efs-mount-helper.html
$ sudo yum install -y amazon-efs-utils (as you are using Amazon Linux, this should already be installed)

3) Make a directory to mount the EFS filesystem:
$ mkdir ~/efs-mount-point

4) Next mount the EFS filesystem using the EFS mount helper:
* https://docs.aws.amazon.com/efs/latest/ug/mounting-fs-mount-helper.html#mounting-fs-mount-helper-ec2
$ sudo mount -t efs fs-b32311b6:/ ~/efs-mount-point/

That should complete the mounting process. Next we can increase the EFS filesystem size by running the following commands: https://aws.amazon.com/premiumsupport/knowledge-center/efs-copy-data-in-parallel/

5) The command below will create a temporary directory in your local storage. It will then create 20 x 1GB files within that directory.
$ mkdir /tmp/efs; for each in $(seq 1 20); do SUFFIX=$(mktemp -u EFS-XXXXXX); sudo dd if=/dev/zero of=/tmp/efs/${SUFFIX} bs=1M count=1024; done

* To confirm the number of files created, you can run the following command:
$ ls -alh /tmp/efs > files_created.txt ; grep -i EFS "files_created.txt" | wc -l

6) Next, run the following commands:
$ sudo amazon-linux-extras install epel
$ sudo yum install nload sysstat parallel -y
$ sudo yum install time -y
$ sudo yum install rsync -y

7) Once you have created the files and installed GNU Parallel, run the following command to copy the files from your instance to the EFS file system:
# didn't work:
$ cd /tmp/efs; time find -L . -maxdepth 1 -type f | sudo parallel rsync -aR {} ~/efs-mount-point

# worked
$ cd /tmp/efs; sudo time find -L . -type f | parallel rsync -avR {} ~/efs-mount-point

Example output while its running:
sent 1,074,004,098 bytes received 35 bytes 49,953,680.60 bytes/sec
total size is 1,073,741,824 speedup is 1.00
sending incremental file list
EFS-UwWFli

sent 1,074,004,099 bytes received 35 bytes 49,953,680.65 bytes/sec
total size is 1,073,741,824 speedup is 1.00
sending incremental file list
EFS-yqR525

sent 1,074,004,099 bytes received 35 bytes 49,953,680.65 bytes/sec
total size is 1,073,741,824 speedup is 1.00
sending incremental file list
EFS-MJjQfR

Once it's completed:
sent 1,074,004,099 bytes received 35 bytes 52,390,445.56 bytes/sec
total size is 1,073,741,824 speedup is 1.00

I was able to confirm my EFS filesystem grew from 6KiB to 20GiB.

EFS - Revision history