Github

Provides docker environment and code which process image. Welcome to star, fork or PR to improve the practice. Repo：Click The Link

What’s this

For the same image link, we may response different sizes of thumbnails according to different pages. Unfortunately, AWS does not directly provide such services. But it provides lambda@edge with CloudFront to handle this kind of demand. Therefore, I designed a set of solutions to achieve the requirements of image processing. This article contains instructions, design solutions and practical steps.

Using this solution, the client can splice parameters behind the original cdn link to obtain the thumbnail, add watermark and other processing requirements.

Roles involved in the solution

CloudFront

Amazon’s CDN service. Be used to cache thumbnails in the solution. Hereinafter referred to as CF

Lambda@edge

Lambda is Amazon’s cloud function, a form of serverless

Lambda@edge is a function that can run on the edge of CloudFront, which is weaker than Lambda. For example, only supports Node.js and Python Runtime; it does not support Layer, so the image library used cannot be deployed on the layer and can only be packaged into the source code.

The image processing logic runs on it, which is the key object of this project. Hereinafter referred to as Lambda

S3

Amazon’s object storage service. Used to store original images, thumbnails, the source which CDN back to.

Workflow

The following is the specific process:

• Step 0: User requests an image link, The link is like: http://cdn.com/image/abc.jpg@250w_60q.webp. Among them, http://cdn.com/image/abc.jpg is the original image link, and those after @ is processing parameters. Please refer to the manual for the specific meaning
  
• Step 1: The request arrives at the CF first, will directly returned to the user if checking that the CF has cache, and response with 304 Not Modified when the etag is same with the client
  
• Step 2: If CF misses, back to the source (S3) to get the image
  
• Step 3: After S3 response, CF will trigger an event to lambda, and the code will check the response status of event:
  • If the http code is a normal value, it means that the thumbnail already exists in S3. So no other processing is needed, return the response object directly
  • Step 5: If the http code is an abnormal value, it means that S3 does not have the image, and we need to do the following:
    • Strip the URI from @, the front is the original image KEY, and the back is the processing parameters. We obtain the original image from the S3 bucket (the bucket name can be extracted from the request link of the CF event property) by that KEY, and then process the image with parameters. We use Sharp Module to process image here (Sharp base on libvips which has better performance than ImageMagic).
    • Put the processed image To S3. Therefore, CF Can back to S3 to get it if CDN cache is expire. By the way, I tagged (createByLambda:1) to the object, so that we can add lifecycle rule with the tags, avoid the waste of too many thumbnails in S3.
    • Response image to CF with base64 encoding. So we need to set content-encoding to base64. The reason why base64 is used is because lambda only supports receiving json
      
• Step4：CF cache the response from origin, and response to client

Parameter description

url format：https://cdn-domain/filename@<value1><param1>_<value2><param2>.outputFormat

example, https://d1xxxxxxxx.cloudfront.net/foamzou/image/4951f0e35a37e5190e78798dcfcad984.jpg@1020w_160h_0e_1l_70q.webp

Parameters

w: Width of thumb. 1-4096

h: Height of thumb. 1-4096

e: Policy about aspect ratio. 0: Keep the aspect ratio, base on long side(Default). 1: Keep the aspect ratio, base on short side. 2: Ignore aspect ratio, force resize with width and height.

l: Whether to process if the target thumbnail is larger than the original image.1: no processing; 0: will processing(Default)

p: Percent of the size (1-1000). Less than 100 means zoom out, and greater than 100 means zoom in. If the parameter p is used in combination with w and h, p will directly act on w and h (multiply by p%) to get the new w and h. For example, 100w_100h_200p has the same effect as 200w_200h. The default value is 100

q: Percent of quality (1-100) . Default: 80

r: Whether to response progressive jpeg. 0: not used (default), 1 used. The default is 0. This parameter will take effect only when the output format is jpg. Note: 1. Small size image are not recommended, because progressive jpeg will be larger than the original image; 2. If you can use webp, try to use webp;

.format: Be output format. Support jpg, png, webp. No need to add .format if you keep the original format

Notice

When the total area exceeds 4096px * 4096px, or the length of one side exceeds 4096px * 4, the image will not be scaled

Practical steps

Create public access bucket in S3

Permissions -> Block public access. Just open the first 2 options:

Bucket Policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicRead",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::your-bucket-name/*"
        }
    ]
}

CORS Configuration

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
    <AllowedOrigin>http://*.your-domain.com</AllowedOrigin>
    <AllowedOrigin>https://*.your-domain.com</AllowedOrigin>
    <AllowedMethod>PUT</AllowedMethod>
    <ExposeHeader>ETag</ExposeHeader>
    <ExposeHeader>x-amz-meta-custom-header</ExposeHeader>
    <AllowedHeader>*</AllowedHeader>
</CORSRule>
</CORSConfiguration>

Use this solution from scratch

Note: Functions must be created in the us-east-1 region

Prepare

• Get the latest code. Repo: Link To Github. Deploy docker according to the readme guide, enter the lambda directory from docker, install dependencies with npm install, compress the entire lambda directory into a zip file(NOTICE: don’t contain directory, you cancd lambda; zip -r ../function.zip ./*; cd -), upload it to S3 (for security, it is recommended to upload to a private bucket), remember the s3 file link, below Deployment needs.
• Create a role to execute lambda. The role need the following permissions（Permissions → Attach policies）
  • AmazonS3FullAccess
  • AWSLambdaReadOnlyAccess
  • AWSLambdaBasicExecutionRole
• Add (Trust relationship → Edit trust relationship)

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    },
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "edgelambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Create Lambda Function

1. Service → Lambda → Functions → Create function, Select Author from scratch
2. Basic information. function name : cdnEdgeImageHandler. Runtime : Node.js 12.x. Execution role → Use an existing role → Select the role just created. Click Create function
3. Upload Code. Open function dashboard, Function Code -> Actions -> Upload a file from Amazon S3. Then paste the zip link (the previous upload) .BTW, even if it is a private bucket link, it does not need to be signed, as long as the bucket belongs to this account, Lambda can access it.
4. Setting runtime. Basic Settings → edit. Handler: index.handler. Memory: change to about 384MB, Timeout set to 15 sec. Then Save.
5. Public new version. Actions → Public new version → write description → Publish. You will see a string(ARN) in the top of the page. Will be use in following. The ARN format: arn:aws:lambda:us-east-1:016655625412:function:cdnEdgeImageHandler:1

Attach Lambda To CloudFront

1. Service -> CloudFront: Create distribution (Skip the step if attach an existed instance). Select a delivery method: Web → Origin Domain Name: select origin which back to → Click Create
2. Create Behavior. Open distribution dashboard → Behavior tab → Create Behavior → Path Pattern: /*.jpg@*   ; Lambda Function Associations: CloudFront Event: Select Origin Response, Function ARN: The string at the top of the function dashboard (NOTICE: the string MUST BE end with version id ) → Click Create
3. If there are other image types, should added as in step 2. Or if you want to uniformly process the pictures under a certain path, you can set a path to Path Pattern. Such as /images/*
4. Add lifecycle rule in S3 bucket (You can choose whether to add the rule). This solution will upload the generated image to S3. In order not to waste space, you can add rules to delete old files. Step: Open S3 dashboard → Management tab → Add lifecycle rule → rule name: Delete cdn cache → rule scope: Limit the scope to specific prefixes or tags → type createByLambda, select tag, type tag value 1. Click next -> Transitions: don’t need to do → next: Configure expiration: Both check Current version and Previous versions. Modify days value → Click Save

How to update Lambda

1. Refer to steps 3 and 5 of “Create Lambda Function” above to release a new version.
2. Lambda Function dashboard → Actions → Deploy to Lambda@edge → Select Use existing CloudFront trigger on this function → Select  CloudFront trigger → Click Deploy
3. Check the effect after a while. Because it takes time for CF to deploy to global nodes. BTW, You should deploy it to the CDN trigger which for testing BEFORE deploy to production.

Skill of Debug

1. Debug it in the development environment first. There is an event.json file in the github, which can be used to simulate CF requests.
2. Before deploying to the edge, use the “test events” in the lambda dashboard to test first. It is recommended to paste the content in event.json as a test event.
3. Make good use of console.log. The log will be report to CloudWatch. But you need to switch to a specific region to see the log of the corresponding request. (The region identification can be seen in the x-amz-cf-pop in the response header of the image, which can be used to locate the region)

Related Information

Demo of Lambda Guide

AWS CDN Blog