EFS

From DikapediaV2
Revision as of 00:10, 27 August 2024 by Ardika Sulistija (talk | contribs) (Created page with "<b>EFS</b> - Elastic Filesystem ====Other methods besides DD==== ---- * How can I copy data to and from Amazon EFS in parallel to maximize performance on my EC2 instance?</br> https://aws.amazon.com/premiumsupport/knowledge-center/efs-copy-data-in-parallel/ DD is not parallel, so there are other methods to create files much quicker than DD. Below is a process on how you can do so: 1) Create ten thousand small files in your local storage on your instance. $ mkdir /t...")
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

EFS - Elastic Filesystem

Other methods besides DD


  • How can I copy data to and from Amazon EFS in parallel to maximize performance on my EC2 instance?

https://aws.amazon.com/premiumsupport/knowledge-center/efs-copy-data-in-parallel/

DD is not parallel, so there are other methods to create files much quicker than DD. Below is a process on how you can do so:

1) Create ten thousand small files in your local storage on your instance.

$ mkdir /tmp/efs; for each in $(seq 1 10000); do SUFFIX=$(mktemp -u EFS-XXXXXX); sudo dd if=/dev/zero of=/tmp/efs/${SUFFIX} bs=64k count=1; done

2) To test copy the files from your instance to the EFS file system:

Serial method: 
$ cd /tmp/efs; time sudo find -L . -maxdepth 1 -type f -exec rsync -aR '{}' /efs/ \;

Parallel method: 
$ cd /tmp/efs; time find -L . -maxdepth 1 -type f | sudo parallel rsync -aR {} /efs/ 

3) To remove:

Serial method: 
$ cd /mnt/efs; time sudo find -L . -maxdepth 1 -exec rm -rfv {} \;

Parallel method: 
$ cd /mnt/efs; find -L . -maxdepth 1 -type f | sudo parallel rm -rfv {}


TESTED BY MOUNTING EFS TO AN INSTANCE, AND CREATED DUMMY FILES IN THE LOCAL STORAGE. THEN USE GNU PARALLEL AND RSYNC TO COPY 20GB of files to my EFS


https://aws.amazon.com/premiumsupport/knowledge-center/efs-copy-data-in-parallel/

1) Create EFS filesystem

2) Connect to your EC2 instance and ensure the amazon-efs-utils package is installed. This package contains the EFS mount helper. The Amazon EFS mount helper simplifies mounting your file systems.

$ sudo yum install -y amazon-efs-utils (as you are using Amazon Linux, this should already be installed)
 

3) Make a directory to mount the EFS filesystem:

$ mkdir ~/efs-mount-point
 

4) Next mount the EFS filesystem using the EFS mount helper:

$  sudo mount -t efs fs-b32311b6:/ ~/efs-mount-point/
 

That should complete the mounting process. Next we can increase the EFS filesystem size by running the following commands: https://aws.amazon.com/premiumsupport/knowledge-center/efs-copy-data-in-parallel/

5) The command below will create a temporary directory in your local storage. It will then create 20 x 1GB files within that directory.

$ mkdir /tmp/efs;  for each in $(seq 1 20); do SUFFIX=$(mktemp -u EFS-XXXXXX); sudo dd if=/dev/zero of=/tmp/efs/${SUFFIX} bs=1M count=1024; done

  • To confirm the number of files created, you can run the following command:
$ ls -alh /tmp/efs > files_created.txt ; grep -i EFS "files_created.txt" | wc -l

6) Next, run the following commands:

$ sudo amazon-linux-extras install epel
$ sudo yum install nload sysstat parallel -y
$ sudo yum install time -y
$ sudo yum install rsync -y

7) Once you have created the files and installed GNU Parallel, run the following command to copy the files from your instance to the EFS file system:

# didn't work:
$ cd /tmp/efs; time find -L . -maxdepth 1 -type f | sudo parallel rsync -aR {} ~/efs-mount-point

# worked
$ cd /tmp/efs; sudo time find -L . -type f | parallel rsync -avR {} ~/efs-mount-point

Example output while its running:

sent 1,074,004,098 bytes  received 35 bytes  49,953,680.60 bytes/sec
total size is 1,073,741,824  speedup is 1.00
sending incremental file list
EFS-UwWFli

sent 1,074,004,099 bytes  received 35 bytes  49,953,680.65 bytes/sec
total size is 1,073,741,824  speedup is 1.00
sending incremental file list
EFS-yqR525

sent 1,074,004,099 bytes  received 35 bytes  49,953,680.65 bytes/sec
total size is 1,073,741,824  speedup is 1.00
sending incremental file list
EFS-MJjQfR

Once it's completed:

sent 1,074,004,099 bytes  received 35 bytes  52,390,445.56 bytes/sec
total size is 1,073,741,824  speedup is 1.00

I was able to confirm my EFS filesystem grew from 6KiB to 20GiB.