This command can also be used to perform sync between S3 Buckets without the need to transit files on a local machine. In this article, we will explore the various options available with the sync command, including the exclude and include options, the delete option, and the dry run option. We will also discuss how to use a specific storage class when uploading files to S3.

#From local to S3
~ aws s3 sync . s3://my-test-bucket

#From S3 to local
~ aws s3 sync s3://my-test-bucket .

# From S3 to S3
~ aws s3 sync s3://my-test-bucket s3://my-copy-test-bucket

Options

Exclude Option

The --exclude option in the AWS CLI sync command can be used to exclude files or objects from the sync based on specific patterns. This option allows users to exclude files based on their file extension, file name, or path. Users can also use the --exclude option multiple times to exclude multiple patterns.

~ aws s3 sync . s3://my-test-bucket --exclude "*.jpg"

Exclude a single large file:

~ aws s3 sync . s3://my-test-bucket --exclude "mydirectory/large-file.tar"

Include Option

In addition, the --exclude option can be used in combination with the --include option to fine-tune file selection. This allows users to include specific files or objects while excluding others based on specific patterns.

aws s3 sync . s3://my-test-bucket --exclude "*" --include ".png" --include ".jpg"

Delete Option

The delete option in the AWS CLI sync command is used to remove files that exist in the destination but not in the source. This is a useful feature for keeping the destination in sync with the source and ensuring that outdated files are removed.

~ aws s3 sync --delete . s3://my-test-bucket
delete: s3://my-test-bucket/testfile1

DryRun Option

The --dryrun option in the AWS CLI sync command allows users to preview the operation that would be executed without actually running it. This is a useful feature for testing and verifying your command before running it to ensure that it will perform the desired action.

~ aws s3 sync --dryrun --delete . s3://my-test-bucket
(dryrun) delete: s3://my-test-bucket/testfile1

Storage Class

The storage class option in the AWS CLI sync command allows users to specify a specific storage class when uploading files to S3.

This can be useful in situations where certain files require a different storage class than the default. For example, if a user needs to store infrequently accessed files, they can use the --storage-class option to specify the STANDARD_IA class.

~ aws s3 sync --storage-class "STANDARD_IA" . s3://my-test-bucket

Introduction

Cloudfront is a Content Distribution Network (CDN), a service that allows you to host your content close to the final users worldwide with low latencies. To know more about CloudFront and CDNs check my article.

S3 is a robust and reliable object storage that will scale on demand and let you only pay for used resources, check it on the AWS page.

Setup React

This setup will work with all static websites but let's focus on React.

So let's start with a simple hello world application created with Create React App https://create-react-app.dev/docs/getting-started

npx create-react-app my-app
cd my-app
npm start

This will create our app and start it, so we can take a look at it in our browser locally on the PC.

Now that we have our sample application ready let's take a look at the structure of the project:

my-app
├── README.md
├── node_modules
├── package.json
├── .gitignore
├── public
│   ├── favicon.ico
│   ├── index.html
│   ├── logo192.png
│   ├── logo512.png
│   ├── manifest.json
│   └── robots.txt
└── src
    ├── App.css
    ├── App.js
    ├── App.test.js
    ├── index.css
    ├── index.js
    ├── logo.svg
    ├── serviceWorker.js
    └── setupTests.js

The structure contains multiple js and media files but those are only the source of our final application, before using it in production we need to build the application and generate an artifact.

We can now kill the React development server we just started and instead we can now build our application:

npm run build

The output of the command will be put in the build directory. All the files inside our build directory are what will be uploaded to our static hosting.

Create S3 resources

With our artifacts ready we can now create an S3 bucket, for this tutorial we will use the AWS console but for your production workloads, I suggest using IaC tools like CDK, Terraform or Cloudformation.

Let's create our bucket:

After creating it, the empty bucket is ready to receive our artifacts:

We can now upload our artifacts, let's do it with the aws-cli, and follow the blog for a future post where we automate this with CI/CD.

aws s3 sync --delete build s3://react-on-aws-demo-alessandromarinoac

Create CloudFront resources

Let's create now our CloudFront distribution and let's select our bucket as origin, using the console make this process really simple.

To secure the connection between CloudFront and the bucket we need to create or use an existing Origin Access Control Settings.

If you don't already have one Control Settings let's create it using the button on the right:

Now with the origin secured, we can start setting up the cache options for our CloudFront distribution, we will for example redirect all HTTP requests to HTTPS.

We can also create a custom cache policy if we need to specify specific TTL or caching options, if not we can use the default CachingOptimized that will help us save some time.

Let's go ahead and scroll to the general settings section, here we can select a price class if we want to save some money or we can for example set our default root object that will be necessary to make React work properly.

We are setting the root object as index.html

We can now click on Create distribution and we have the confirmation. The distribution needs some time to be ready (~10min) so have some patience here.

Secure our S3 Bucket

Now as the console advise us to, let's copy the S3 bucket policy from the blue banner and let's go back to our S3 bucket to apply it. We need to navigate to the bucket and then to the Permission tab:

We can now apply the copied S3 Bucket Policy to restrict access to the bucket:

With this policy in place, we are authorizing CloudFront to read from our bucket, without the need of having a public bucket.

Check if it's working

Let's now go back to the CloudFront console and retrieve the distribution domain name to check our app online:

If we use the browser to open that link, we can see our app:

So now we have our React application up and running.

Setup Errors Pages

There is still a step we need to take. If we try to navigate to any page other than the root we will get an error:

We need to set Error Pages on CloudFront so instead of getting errors from CloudFront we will let React take care of those.

Let's create two custom error responses, one for 403 Forbidden and one for 404 Not Found , both will have a response code of 200 OK.

With these new settings, we can now navigate to every path of our applications without receiving errors from CloudFront:

To create pages on different paths inside react we will need to use something like https://reactrouter.com/en/main

With react-router configured we can also set up custom 404 pages.

Why you could need large-scale parallel data processing?

Processing large CSVs or processing every item in an S3 Bucket is something that can be necessary from time to time.
Some examples use cases:

1. Import data from a large CSV file in a short time

2. Create many ordered requests to third-party service with retry and backoff, where each request uses the output from the precedent.

3. Reprocess every image you already have on your S3 bucket to follow a new standard size

Why use serverless resources for it?

Processing millions of items with a tight deadline in a traditional way, using containers or virtual machines, can need quite some effort of pre-provisioning. If our input doesn't come at a predictable time this can be even harder.
By using a combination of services like Step Function and Lambda we can run our workflow without pre-provisioning, run it when needed, and pay only for the usage.

Make sure to verify your AWS Service Limits if you plan to run a large number of Lambda.

Demo

I built a small demo using AWS CDK to run an image resizing process on an S3 bucket containing 8088 images of dogs. The project uses AWS CDK (typescript) and the lambda function executing the processing is built with Go.

The GitHub repository
The dataset used for testing

The Step Function

The step function for this demo is really simple:

{
    "Comment": "A description of my state machine",
    "StartAt": "Map",
    "States": {
        "Map": {
            "Type": "Map",
            "ItemProcessor": {
                "ProcessorConfig": {
                    "Mode": "DISTRIBUTED",
                    "ExecutionType": "EXPRESS"
                },
                "StartAt": "Lambda Invoke",
                "States": {
                    "Lambda Invoke": {
                        "Type": "Task",
                        "Resource": "arn:aws:states:::lambda:invoke",
                        "OutputPath": "$.Payload",
                        "Parameters": {
                            "Payload.$": "$",
                            "FunctionName.$": "$.BatchInput.lambda_processor_arn"
                        },
                        "Retry": [
                            {
                                "ErrorEquals": [
                                    "Lambda.ServiceException",
                                    "Lambda.AWSLambdaException",
                                    "Lambda.SdkClientException",
                                    "Lambda.TooManyRequestsException"
                                ],
                                "IntervalSeconds": 2,
                                "MaxAttempts": 6,
                                "BackoffRate": 2
                            }
                        ],
                        "End": true
                    }
                }
            },
            "End": true,
            "Label": "Map",
            "MaxConcurrency": 1000,
            "ItemReader": {
                "Resource": "arn:aws:states:::s3:listObjectsV2",
                "Parameters": {
                    "Bucket.$": "$.input.source_bucket_name",
                    "Prefix.$": "$.input.bucket_path"
                }
            },
            "ItemBatcher": {
                "MaxItemsPerBatch": 100,
                "BatchInput": {
                    "lambda_processor_arn.$": "$.input.lambda_processor_arn",
                    "source_bucket_name.$": "$.input.source_bucket_name",
                    "destination_bucket_name.$": "$.input.destination_bucket_name"
                }
            }
        }
    }
}

We make use of the freshly released DISTRIBUTED mode of the Map state and we create a single Lambda function as a child of this Map state.

We have multiple possible options to use as a source for our distributed processing, in this case, we are using arn:aws:states:::s3:listObjectsV2 that will perform a list operation on the s3 path provided and it will trigger a child execution for each file found (or for a batch in this case, see ItemBatcher item).

Other than the batch settings (the number of items that will go in each child execution) we can also set how many parallel child executions we want with the MaxConcurrency parameter.

In this example I'm setting a batch of 100 items with a max concurrency of 1000 execution, so theoretically the 8088 files will be executed by 81 Lambda functions.

Results

If we run our Step Function we can see the events that take place:

And if we take a look at the Map Run dashboard we can see the details about the child executions of the Step Function:

With these batch settings, we run the resizing process on 8088 photos in just 8.4 seconds, and if we take a look at Lambda metrics we can see that Lambda duration has an average value of 2.3 seconds and we have exactly 81 invocations with 81 concurrent executions, so exactly what we were expecting.

I also did run an experiment with a batch size of 10 and 809 invocations were triggered with a peak of 341 concurrent execution and an average duration of 432ms.

Our resized dog:

CloudFront is Amazon’s offer for Content Delivery Network. The CDN is composed of a globally distributed set of caching servers that provide low latency and high throughput all around the world.

The network is based on more than 300 PoP (Points of Presence) around the world.
These PoPs are for majority edge locations that serve traffic to the final users but, usually corresponding to AWS regions, there are also regional edge locations. This latter one can be considered a mid-tier cache.

Image from AWS Docs

Distributions

A CloudFront distribution is the main resource you can create to distribute content globally, for example when creating a WordPress site we can create a distribution for WordPress media. One of the main settings that a Distribution allows you to specify it’s the content origin, this origin is where CloudFront will try to fetch content before caching it.

Origins

CloudFront supports multiple origins and natively integrates with many AWS Services.
One of the most used integration is with Amazon S3. The combination of S3 + CloudFront it’s widely used to host static websites.
When using CloudFront with Amazon S3 it’s possible to secure the origin’s bucket with an Origin Access Identity (OAI), this feature will protect the bucket from traffic that it’s not originated within your CloudFront distribution.

Caching Behaviour

CloudFront allows you to customize how requests are handled with multiple cache behavior. Each cache behavior allows the user to define a pattern (eg: *.jpg) and some rules that apply to this pattern.

Static websites provide a good example of how to use cache behaviors. All the media files (JPG, PNG, SVG) are generally cached for a long time meanwhile index.html is generally cached for a shorter time.
So by having multiple cache behaviors we can have all these customized settings for each file type.

Functions

Lambda@Edge

These Lambda functions allow to run code near the end user, specifically, Lambda@Edge are executed in a regional edge location.

Lambda@Edge can be considered normal Lambda functions with some more limitations on execution times, runtimes, and other settings. Some actions that are commonly performed in Lambda@Edge:

• connect to a database

• execute network calls (eg: HTTP)

Lambda@Edge are invoked by CloudFront on specific events:

1. Viewer Request

2. Origin Request

3. Origin Response

4. Viewer Response

So for example by using the event “Origin Request” you can trigger a Lambda@Edge every time CloudFront contacts your origin (every cache miss).

CloudFront Functions

CloudFront functions are lightweight functions that are executed at the edge, so in one of the 300+ PoP. Instead, Lambda@Edge are executed in regional edge location (13 at the time of writing).

CloudFront functions have low latency but also come with some hard limitations on execution time and maximum memory:

• Maximum execution time: 1ms

• Maximum memory: 2MB

Also, CloudFront functions come without network or filesystem access. Common uses for CloudFront functions are:

• URL rewrite and redirects

• Cache-key manipulations and normalizations

• HTTP header manipulation

• Access Authorisation

CloudFront functions are invoked by CloudFront on specific events (less than Lambda@Edge):

1. Viewer Request

2. Viewer Response

Differences recap

GeoRestriction

CloudFront allows the user to block certain geographical zones from accessing the content. To make an example of the use of this feature: streaming services block the viewing of some content in regions where copyright it’s held by other companies.

Pricing

When using CloudFront with AWS services you pay only for outbound traffic: from CloudFront to the final users. Traffic from AWS service to CloudFront it’s free.

Pricing on AWS services tends to change often so please refer to https://aws.amazon.com/cloudfront/pricing

Free Tier

CloudFront has recently updated its free tier, starting from Dec 1, 2021, the new free tier it’s not limited to 12 months after signup but extends indefinitely. New limits are:

• Data Traffic: 1TB/month

• Requests: 10M/month

What Terraform Workspaces are?

Terraform supports two concepts of workspaces: CLI workspace and Terraform Cloud/Enterprise workspace. This article is focused on CLI Workspaces also known as OSS Workspace.

CLI workspaces are a feature that allows us to manage a single Terraform configuration and provision the resulting resources multiple times.
This allows us to have similar groups of resources without managing multiple stacks. To use CLI workspaces, we have to use a compatible backend, like S3.

From this point, I will call CLI Workspaces as workspaces.

Use Case

Let's imagine a simple infrastructure composed of some virtual machines deployed in three different environments:

1. Development

2. Staging

3. Production

Each one of those three environments is on a dedicated AWS account, and they don't share anything. Configuration files for the environments are the same and must remain consistent and identical.

Differences like the environment tags, instance size, instance number, and other parameters are given with a dedicated variable file tfvars.

We usually have two ways to deploy this configuration multiple times:

• different directories

• workspaces

Different directories

This approach guarantees the optimal separation between environments and helps reduce errors, like deploying to the wrong environment.

Having multiple directories also implies that the same code is replicated in all of them, so this is a big downside but can be also an advantage for use cases where heavy customization between environments is required.

In our case, wanting three identical environments, this approach will only be more complex and harder to maintain.

CLI Workspaces

Workspaces are usually my way to resolve this issue and avoid replicated code. Workspaces as a "copy" of the state file that relays on the same terraform configuration files, so you can write once and deploy several times the exact replica of your infrastructure.

When working with workspaces, we manage multiple replicas of the deployed infrastructure from the same configuration file, in a single directory. Working in this single directory can introduce some human errors like the deploy in the wrong environment or destroying the wrong environment.

Workspace setup

This is a small example with S3 as a remote backend. This example will be deployed on three different AWS accounts, one for each environment:

• dev

• stage

• prod

Initial setup (Backend)

Using S3 as our remote backend our Terraform state will be saved on Amazon S3. To maintain all our Terraform states in a single place we choose to use our production account as storage. So we need to create an S3 bucket and a DynamoDB table on our production account, the bucket for this example will be named my-terraform-backend-state. Set up S3 Backend

Initial setup (CLI)

Before applying the terraform configuration we will have to set up the AWS CLI.
Configuring the AWS CLI

With the AWS CLI installed we can set up our AWS Profile: AWS CLI Named Profiles

~/.aws/credentials

[my-account-dev]
aws_access_key_id=********************
aws_secret_access_key==******************

[my-account-stage]
aws_access_key_id=********************
aws_secret_access_key==******************

[my-account-prod]
aws_access_key_id=********************
aws_secret_access_key==******************

The Terraform Configuration

provider "aws" {
  region  = var.aws_region
  profile = "my-account-${terraform.workspace}"
}

terraform {
  required_version = "= 0.15.4"

   required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.0"
    }
  }

  backend "s3" {
    profile              = "my-account-prod"
    bucket               = "my-terraform-backend-state"
    workspace_key_prefix = "eu-west-1/tutorial"
    key                  = "shared-infra"
    region               = "eu-west-1"

    dynamodb_table = "terraform-states-lock-table"
  }
}

This configuration hides a lot of details, so we have to deep dive a little bit into each one.

Profiles

Profiles are used by the Terraform AWS Provider to authenticate by the use of the named profile we set up early. The complex part here is that we use the profile parameter two times. The first use of profile is inside the provider block, in this case, the parameter is used to deploy the resources we declare in our stack. So if we have to deploy the stack in our development account the profile here must be my-account-dev and by using

profile = "my-account-${terraform.workspace}"

we interpolate our current workspace and in case we are deploying to the development account the workspace name will exactly be dev. So the profile after interpolation will be: my-account-dev, exactly what we need.

The second use of profile is inside the backend block, here we use the parameter to indicate Terraform where our backend (S3 Bucket + DynamoDB Table) will be. In our case, we choose to centralize states and the DynamoDB lock table on the production account so the profile will be my-account-prod.

Deployment

Workspaces Commands

1. Creating a new workspace:

terraform workspace new dev

1. Select an existing workspace:

terraform workspace select dev

1. Apply using a specific var file

terraform apply -var-file=terraform.dev.tfvars

Resulting State Files

The example below if deployed to all our three accounts will produce three different state files.

s3://my-terraform-backend-state/eu-west-1/tutorial/dev/shared-infra
s3://my-terraform-backend-state/eu-west-1/tutorial/stage/shared-infra
s3://my-terraform-backend-state/eu-west-1/tutorial/prod/shared-infra

This state file is composed by:

s3://{bucket_name}/{workspace_key_prefix}/{workspace_name}/{key}.json

List all buckets

~ aws s3 ls
2019-12-15 16:31:53 testbucket1
2020-03-11 14:44:32 testbucket112312031230
2020-06-01 09:06:26 myPersonalTestBucket

List objects inside a bucket

~ aws s3 ls s3://myPersonalTestBucket/
                           PRE 2020/
2019-12-15 18:05:39      69483 test.PNG

When ls it's used inside a bucket, it will return all objects and prefixes inside the current level. You can navigate prefixes as a sort of virtual directory.
In this case, we can also navigate down inside the 2020 prefix:

~ aws s3 ls s3://myPersonalTestBucket/2020/09/01/20/20/
2019-12-15 18:23:46   32864363 complete-logs.txt
2019-12-15 18:23:46     463004 light-logs.pdf

Path matching

Let's take into consideration a bucket with content similar to this one:

~ aws s3 ls s3://testbucket1
                           PRE ABC/
                           PRE ACC/
                           PRE ACZ/
                           PRE AZC/
                           PRE BCA/
                           PRE BCZ/

What if we want to list all prefixes that start with A or AC:

~ aws s3 ls s3://testbucket1/A
                           PRE ABC/
                           PRE ACC/
                           PRE ACZ/
                           PRE AZC/
~ aws s3 ls s3://testbucket1/AC
                           PRE ACC/
                           PRE ACZ/

And what if we want to list all objects starting with a prefix:

~ aws s3 ls s3://testbucket1/AC --recursive
2019-12-15 18:22:17     99731 ACC/test1.pdf
2019-12-15 18:22:17     1916928 ACZ/test2.pdf
2019-12-15 18:22:17     112989 ACZ/test3.pdf

Human option

The human option helps read the filesize.

~ aws s3 ls s3://testbucket1/AC --recursive --human
2019-12-15 18:22:17     97.4 KiB ACC/test1.pdf
2019-12-15 18:22:17     1.8 MiB ACZ/test2.pdf
2019-12-15 18:22:17     110.3 KiB ACZ/test3.pdf

Summarize option

Summarize option add two rows at the end of the output with a short recap:

• total objects listed

• the total size of objects listed

You can use --summarize in combination with --human to have the total size of objects listed a little more readable.

~ aws s3 ls s3://testbucket1/AC --recursive --human --summarize
2019-12-15 18:22:17     97.4 KiB ACC/test1.pdf
2019-12-15 18:22:17     1.8 MiB ACZ/test2.pdf
2019-12-15 18:22:17     110.3 KiB ACZ/test3.pdf
Total Objects: 3
   Total Size: 2.1 MiB

• Using Homebrew

• Manual Installation from binary

• Using TFSwitch

Official Terraform documentation.

Using TFSwitch

I will start with the third method because I like the tool. TFSwitch is an open-source project that allows you to easily switch Terraform's version. TFSwitch is created and maintained by warrensbox, here the GitHub repo.
Before starting make sure to uninstall Terraform if you have it already installed.

To install TFSwitch we can use homebrew:

brew install warrensbox/tap/tfswitch

Using Homebrew

If you want to stick to the classical Terraform installation, you can still use homebrew:

brew install hashicorp/tap/terraform

You can verify your current version with:

terraform -v

Manual Install

Terraform is distributed as a single binary, you can find the latest version on Terraform website.
After downloading Terraform we can simply put the binary in a directory listed in PATH.

mv ~/Downloads/terraform /usr/local/bin/

To check if the installation is working properly:

terraform -v

If you are on MacOS or Linux there is a really useful tool written by warrensbox called tfswitch. You can find it on GitHub.

tfswitch allows you to easily switch between Terraform versions, and if you don't have the selected version yet, it will download and install it for you.

Installation

MacOS

You can install tfswitch with homebrew:

brew install warrensbox/tap/tfswitch

Linux

You can install tfswitch with curl script:

curl -L https://raw.githubusercontent.com/warrensbox/terraform-switcher/release/install.sh | bash

min

min(set number) number

Given a set of numbers (or a compatible structure), this function returns the smallest one

>min(1,55,127,4,10)
1

>min([55,4,10]...)
4

max

max(set number) number

Given a set of numbers (or a compatible structure), this function returns the biggest one

>max(1,55,127,4,10)
127

>max([55,4,10]...)
55

parseint

parseint(string stringNumber, number base) number

Given a string representation of a number and its base, this function will return an integer.
The base must be between 2 and 62 inclusive.

>parseint("25", 10)
25

>parseint("1FF", 16)
511

>parseint("111", 2)
7

floor

floor(number num)

This function will return the closest whole number that is less or equal to the input number. The input number can be decimal.

>floor(11.9)
11

>floor(11)
11

floor(9.1)
9

ceil

ceil(number num)

This function will return the closest whole number that is greater or equal to the input number. The input number can be decimal.

>ceil(11.9)
12

>floor(11)
11

floor(9.1)
10

abs

abs(number num) number

Given an integer number, abs will return its absolute value. If the input number is negative, it will be multiplied by -1.

>abs(10)
10

>abs(-10)
10

>abs(0)
0

pow

pow(number num, number elevation)

This function will elevate the first argument num to the second argument elevation.

>pow(4,2)
16

>pow(3,1)
3

>pow(9,0)
1

signum

signum(number num)

This function returns a number representing the symbol of the input number.

>signum(-1)
-1

>signum(1)
1

>signum(9999)
1

>signum(0)
0

What are multi-stage builds?

Multi-stage builds are a feature Docker has implemented to offer a better and cleaner organization of the Dockerfile. This is achieved using multiple sections inside a single Dockerfile, those sections are called stages.
Each stage allows you to use a different FROM statement (a different base image) and selectively copy artifacts from one stage to another.
When you build a Dockerfile containing multiple stages, Docker will build all those different stages separately. Before multi-stage builds the only way to achieve this, was to use, and maintain, different Docker files.

How we can use a multi-stage build to reduce image size?

This feature allows you to bring your own artifacts between stages, so you can easily build your application in a different stage than the one that would run it.

Why this is an advantage?

When building an application we need to install a lot of development dependencies, which maybe are not required to run the resulting artifacts. So we can easily discard all of them.
Surely you can try to remove all of those dependencies with some inverse commands but I will show you how simpler is to do that with multi-stage builds. An example can be packages installed with a package manager like apk or apt, which usually brings with them a lot of dependencies.

Dockerfile

FROM golang:1.13-alpine3.10 AS build
RUN apk --no-cache add git
WORKDIR /app
COPY . .
ENV GO111MODULE=on
RUN go mod download
RUN GOOS=linux go build -ldflags="-s -w" -o ./test ./main.go
CMD ["/app/test"]

FROM scratch
WORKDIR /
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=build /app/test /app
CMD ["/app"]

main.go

package main

import "log"

func main() {
    log.Println("Test")
}

How it works

In this multi-stage Dockerfile, we build our Go package in a stage called build and we get the resulting executable in a clean stage based on a scratch image.
So the actual image produced is only the size of the base image (scratch) and the executable that, in this case, is only some KB large.

Results

• Image produced with multi-stage build: 1.74MB

• Image produced with classical single stage build from Go image: 79MB

With some optimization, the classic build size can also be reduced, example of those optimizations are cache cleaning, and file removal...
Even with those optimizations, the size difference is huge and definitively notable.