The AWS CLI sync command is a powerful tool for copying new and updated files between a source and destination, with at least one of them being an S3 Bucket.

This command can also be used to perform sync between S3 Buckets without the need to transit files on a local machine. In this article, we will explore the various options available with the sync command, including the exclude and include options, the delete option, and the dry run option. We will also discuss how to use a specific storage class when uploading files to S3.

#From local to S3
~ aws s3 sync . s3://my-test-bucket

#From S3 to local
~ aws s3 sync s3://my-test-bucket .

# From S3 to S3
~ aws s3 sync s3://my-test-bucket s3://my-copy-test-bucket

Options

Exclude Option

The --exclude option in the AWS CLI sync command can be used to exclude files or objects from the sync based on specific patterns. This option allows users to exclude files based on their file extension, file name, or path. Users can also use the --exclude option multiple times to exclude multiple patterns.

~ aws s3 sync . s3://my-test-bucket --exclude "*.jpg"

Exclude a single large file:

~ aws s3 sync . s3://my-test-bucket --exclude "mydirectory/large-file.tar"

Include Option

In addition, the --exclude option can be used in combination with the --include option to fine-tune file selection. This allows users to include specific files or objects while excluding others based on specific patterns.

aws s3 sync . s3://my-test-bucket --exclude "*" --include ".png" --include ".jpg"

Delete Option

The delete option in the AWS CLI sync command is used to remove files that exist in the destination but not in the source. This is a useful feature for keeping the destination in sync with the source and ensuring that outdated files are removed.

~ aws s3 sync --delete . s3://my-test-bucket
delete: s3://my-test-bucket/testfile1

DryRun Option

The --dryrun option in the AWS CLI sync command allows users to preview the operation that would be executed without actually running it. This is a useful feature for testing and verifying your command before running it to ensure that it will perform the desired action.

~ aws s3 sync --dryrun --delete . s3://my-test-bucket
(dryrun) delete: s3://my-test-bucket/testfile1

Storage Class

The storage class option in the AWS CLI sync command allows users to specify a specific storage class when uploading files to S3.

This can be useful in situations where certain files require a different storage class than the default. For example, if a user needs to store infrequently accessed files, they can use the --storage-class option to specify the STANDARD_IA class.

~ aws s3 sync --storage-class "STANDARD_IA" . s3://my-test-bucket

AWS CLI S3 Cheatsheet - s3 sync

Comments

AWS-CLI Cheat Sheets

AWS CLI S3 Cheatsheet - s3 ls

More from this blog

Boosting Monolith Performance: Caching with OpenResty, SRCache, and Redis

Deploy a React application on AWS with CloudFront and S3

Unleashing the Power of Large-Scale Parallel Data Processing on AWS

Introduction to Amazon CloudFront

Options

Exclude Option

Include Option

Delete Option

DryRun Option

Storage Class

Command Palette

Comments

AWS-CLI Cheat Sheets

AWS CLI S3 Cheatsheet - s3 ls

More from this blog

Options

Exclude Option

Include Option

Delete Option

DryRun Option

Storage Class