Tim Wagner, AWS Lambda General Manager

Quick, what’s wrong with the Lambda code sketch below?

exports.handler = function(event, context) {
    anyAsyncCall(args, function(err, result) {
        if (err) console.log('problem');
        else /* do something with result */;
    });
    context.succeed();
};

If you said the placement of context.succeed, you’re correct – it belongs inside the callback. In general, when you get this wrong your code exits prematurely, after the incorrectly placed context.succeed line, without allowing the callback to run. The same thing happens if you make calls in a loop, often leading to race conditions where some callbacks get “dropped”; the lack of a barrier synchronization forces a too-early exit.

If you test outside of Lambda, these patterns work fine in nodejs, because the default node runtime waits for all tasks to complete before exiting. Context.succeed, context.done, and context.fail however, are more than just bookkeeping – they cause the request to return after the current task completes, even if other tasks remain in the queue. Generally that’s not what you want if those tasks represent incomplete callbacks.

Placement Patterns

Fixing the code in the single callback case is trivial; the code above becomes

exports.handler = function(event, context) {
    anyAsyncCall(args, function(err, result) {
        if (err) console.log('problem');
        else /* do something with result */;
        context.succeed();
    });
};

Dealing with a loop that has an unbounded number of async calls inside it takes more work; here’s one pattern (the asyncAll function) used in Lambda’s test harness blueprint to run a test a given number of iterations:

/**
 * Provides a simple framework for conducting various tests of your Lambda
 * functions. Make sure to include permissions for `lambda:InvokeFunction`
 * and `dynamodb:PutItem` in your execution role!
 */
var AWS = require('aws-sdk');
var doc = require('dynamodb-doc');

var lambda = new AWS.Lambda({ apiVersion: '2015-03-31' });
var dynamo = new doc.DynamoDB();


// Runs a given function X times
var asyncAll = function(opts) {
    var i = -1;
    var next = function() {
        i++;
        if (i === opts.times) {
            opts.done();
            return;
        }
        opts.fn(next, i);
    };
    next();
};


/**
 * Will invoke the given function and write its result to the DynamoDB table
 * `event.resultsTable`. This table must have a hash key string of "testId"
 * and range key number of "iteration". Specify a unique `event.testId` to
 * differentiate each unit test run.
 */
var unit = function(event, context) {
    var lambdaParams = {
        FunctionName: event.function,
        Payload: JSON.stringify(event.event)
    };
    lambda.invoke(lambdaParams, function(err, data) {
        if (err) {
            context.fail(err);
        }
        // Write result to Dynamo
        var dynamoParams = {
            TableName: event.resultsTable,
            Item: {
                testId: event.testId,
                iteration: event.iteration || 0,
                result: data.Payload,
                passed: !JSON.parse(data.Payload).hasOwnProperty('errorMessage')
            }
        };
        dynamo.putItem(dynamoParams, context.done);
    });
};

/**
 * Will invoke the given function asynchronously `event.iterations` times.
 */
var load = function(event, context) {
    var payload = event.event;
    asyncAll({
        times: event.iterations,
        fn: function(next, i) {
            payload.iteration = i;
            var lambdaParams = {
                FunctionName: event.function,
                InvocationType: 'Event',
                Payload: JSON.stringify(payload)
            };
            lambda.invoke(lambdaParams, function(err, data) {
                next();
            });
        },
        done: function() {
            context.succeed('Load test complete');
        }
    });
};


var ops = {
    unit: unit,
    load: load
};

/**
 * Pass the test type (currently either "unit" or "load") as `event.operation`,
 * the name of the Lambda function to test as `event.function`, and the event
 * to invoke this function with as `event.event`.
 *
 * See the individual test methods above for more information about each
 * test type.
 */
exports.handler = function(event, context) {
    if (ops.hasOwnProperty(event.operation)) {
        ops[event.operation](event, context);
    } else {
        context.fail('Unrecognized operation "' + event.operation + '"');
    }
};

The approach above serializes the loop; there are many other approaches, and you can use async or other libraries to help.

Does this matter for Java or other jvm-based languages in AWS Lambda?

The specific issue discussed here – the “side effect” of the placement of a call like context.success on outstanding callbacks – is unique to nodejs. In other languages, such as Java, returning from the thread of control that represents the request ends the request, which is a little easier to reason about and generally matches developer expectations. Any other threads or processes running at the time that request returns get frozen until the next request (assuming the container gets reused; i.e., possibly never), so if you want them to wrap up, you would need to include explicit barrier synchronization before returning, just as you normally would for a server-side request implemented with multiple threads/processes.

In all languages, context also offers useful “environmental” information (like the request id) and methods (like the amount of time remaining).

Why not just let nodejs exit, if its default behavior is fine?

That would require every request to “boot” the runtime…potentially ok in a one-off functional test, but latency would suffer and it would keep high request rates from being cost effective. Check out the post on container reuse for more on this topic.

Can you make this easier?

You bet! We were trying hard to balance simplicity, control, and a self-imposed “don’t hack node” rule when we launched back in November. Fortunately, newer versions of nodejs offer more control over “exiting” behavior, and we’re looking hard at how to make future releases of nodejs within Lambda offer easier to understand semantics without losing the latency and cost benefits of container reuse. Stay tuned!

Until next time, happy Lambda (and contextually successful nodejs) coding!

-Tim
Follow Tim’s Lambda adventures on Twitter

My colleague Constantin Gonzalez sent a nice guest post that shows how to create container workers using Amazon ECS.

—

Amazon EC2 Container Service (Amazon ECS) is a highly scalable, high performance container management service that supports Docker containers and allows you to easily run applications on a managed cluster of Amazon EC2 instances. ECS eliminates the need for you to install, operate, and scale your own cluster management infrastructure.

AWS Lambda is a compute service that runs your code in response to events and automatically manages the compute resources for you, making it easy to build applications that respond quickly to new information. Lambda starts running your code within milliseconds of an event such as an image upload, in-app activity, website click, or output from a connected device.

In this post, we show you how to combine the two services to mutually enhance their capabilities: See how you can get more out of Lambda by using it to start ECS tasks and how you can turn your ECS cluster into a dynamic fleet of container workers that react to any event supported by Lambda.

Example Setup: Ray-tracing high-quality images in the cloud

To illustrate this pattern, you build a simple architecture that generates high-quality, ray-traced images out of input files written in a popular, open-source raytracing language called POV-Ray, conveyed by POV and licensed under either POV’s proprietary license (up to version 3.6) or AGPLv3 (version 3.7 onwards). Here’s an overview of the architecture:

To use this architecture, put your POV-Ray scene description file (a POV-Ray .POV file) and its rendering parameters (a POV-Ray .INI file), as well as any supporting other files (e.g., texture images), into a single .ZIP file and upload it to an Amazon S3 bucket. In this architecture, the bucket is configured with an S3 event notification which triggers a Lambda function as soon as the .ZIP file is uploaded.

In similar setups (such as transcoding images), Lambda alone would be sufficient to perform its job on the uploaded object within its allocated time frame (currently 60 seconds). But in this example, we want to support complex rendering jobs that usually take significantly longer.

Therefore, the Lambda function simply takes the event data it received from S3 and sends it as a message into an Amazon Simple Queue Service (SQS) queue. SQS is a fast, reliable, scalable, fully-managed message queuing service. The Lambda function then starts an ECS task that can fetch and process the message from SQS.

Your ECS task contains a simple shell script that reads messages from SQS, extracts the S3 bucket and key information, downloads and unpacks the .ZIP file from S3 and proceeds to starting POV-Ray with the downloaded scene description and data.

After POV-Ray has performed its rendering magic, the script takes the resulting .PNG picture and uploads it back to the same S3 bucket where the original scene description was downloaded from. Then it deletes the message from the queue to avoid duplicate message processing.

The script continues pulling, processing, and deleting messages from the SQS queue until it is fully drained, then it exits, thereby terminating its own container.

Simple and efficient event-driven computing

This architecture can help you:

• Extend the capabilities of Lambda to support any processing time, more programming languages, or other resource requirements, to take advantage of the flexibility of Docker containers.
• Extend the capabilities of ECS to allow event-driven execution of ECS tasks: Use any event type supported by Lambda to start new ECS tasks for processing events, run long batch jobs triggered by new data in S3, or any other event-driven mechanism that you want to implement as a Docker container.
• Get the best of both worlds by coupling the dynamic, event-driven Lambda model with the power of the Docker eco-system.

Step-by-Step

Sounds interesting? Then get started!

This is an advanced architecture example covering a number of AWS services like Lambda, ECS, S3, and SQS in depth as well as using multiple related IAM policies. The underlying resources are in your account and subject to their pricing.

To make it easier for you to follow, we have published all necessary code and scripts on GitHub in the awslabs/lambda-ecs-worker-pattern repository. You might find it even more helpful if you could become familiar with the mentioned services by working through the respective Getting Started documentation first.

Meet the following prerequisites:

• Pick an AWS region that support Lambda, ECS, and SQS.
• Set up the AWS CLI. For more information, see the Setting Up topic in the ECS documentation.
• Get familiar with running Lambda functions and how to trigger them from S3 event notifications. For more information, see Getting Started 2: Handling Amazon S3 Events Using the AWS Lambda Console (Node.js).
• Start a default ECS cluster. For more information, see Getting Started with Amazon ECS.
• Become familiar with Docker files, creating Docker images, and using DockerHub. For more information, see the Docker User Guide.
• Set up a DockerHub account for free or use a Docker repository of your own. This demo uses DockerHub to manage and store Docker images.

This post walks through the steps required for setup, then gives you a simple Python script that can perform all the steps for you.

Step 1: Set up an S3 bucket
Start by setting up an S3 bucket to hold both the POV-Ray input files and the resulting .PNG output pictures. Choose a bucket name and create it:

$ aws s3 mb s3://

Step 2: Create an SQS queue
Use SQS to pass the S3 notification event data from Lambda to your ECS task. You can create a new SQS queue using the following command:

$ aws sqs create-queue --queue-name ECSPOVRayWorkerQueue

Step 3: Create the Lambda function
The following function reads in a configuration file with the name of an SQS queue, an ECS task definition name, and a whitelist of accepted input file types (.ZIP, in this example).

The config file uses JSON and looks like this (make sure to use your region):

$ cat ecs-worker-launcher/config.js
{
    "queue": "https://<YOUR-REGION>.queue.amazonaws.com/<YOUR-AWS-ACCOUNT-ID>/ECSPOVRayWorkerQueue",
    "task": "ECSPOVRayWorkerTask",
    "s3_key_suffix_whitelist": [".zip"]
}

The SQS queue ARN (which looks like: https://eu-west-1.queue.amazonaws.com/<YOUR-AWS-ACCOUNT-ID>/ECSPOVRayWorkerQueue) is the output of the preceding command, in which you created your ECS queue. The “task” attribute references the name of an ECS task that you create in a future step.

The Lambda function checks the S3 object key given in the Lambda event against the file type whitelist; in case of a match, it sends a message to the configured SQS queue with the event data and starts the ECS task specified in the configuration file. Here’s the code:

// Copyright 2015 Amazon.com, Inc. or its affiliates. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License").
// You may not use this file except in compliance with the License.
// A copy of the License is located at
//
//    http://aws.amazon.com/apache2.0/
//
// or in the "license" file accompanying this file.
// This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and limitations under the License.

// This Lambda function forwards the given event data into an SQS queue, then starts an ECS task to
// process that event.

var fs = require('fs');
var async = require('async');
var aws = require('aws-sdk');
var sqs = new aws.SQS({apiVersion: '2012-11-05'});
var ecs = new aws.ECS({apiVersion: '2014-11-13'});

// Check if the given key suffix matches a suffix in the whitelist. Return true if it matches, false otherwise.
exports.checkS3SuffixWhitelist = function(key, whitelist) {
    if(!whitelist){ return true; }
    if(typeof whitelist == 'string'){ return key.match(whitelist + '$') }
    if(Object.prototype.toString.call(whitelist) === '[object Array]') {
        for(var i = 0; i < whitelist.length; i++) {
            if(key.match(whitelist[i] + '$')) { return true; }
        }
        return false;
    }
    console.log(
        'Unsupported whitelist type (' + Object.prototype.toString.call(whitelist) +
        ') for: ' + JSON.stringify(whitelist)
    );
    return false;
};

exports.handler = function(event, context) {
    console.log('Received event:');
    console.log(JSON.stringify(event, null, '  '));

    var config = JSON.parse(fs.readFileSync('config.json', 'utf8'));
    if(!config.hasOwnProperty('s3_key_suffix_whitelist')) {
        config.s3_key_suffix_whitelist = false;
    }
    console.log('Config: ' + JSON.stringify(config));

    var key = event.Records[0].s3.object.key;

    if(!exports.checkS3SuffixWhitelist(key, config.s3_key_suffix_whitelist)) {
        context.fail('Suffix for key: ' + key + ' is not in the whitelist')
    }

    // We can now go on. Put the S3 URL into SQS and start an ECS task to process it.
    async.waterfall([
            function (next) {
                var params = {
                    MessageBody: JSON.stringify(event),
                    QueueUrl: config.queue
                };
                sqs.sendMessage(params, function (err, data) {
                    if (err) { console.warn('Error while sending message: ' + err); }
                    else { console.info('Message sent, ID: ' + data.MessageId); }
                    next(err);
                });
            },
            function (next) {
                // Starts an ECS task to work through the feeds.
                var params = {
                    taskDefinition: config.task,
                    count: 1
                };
                ecs.runTask(params, function (err, data) {
                    if (err) { console.warn('error: ', "Error while starting task: " + err); }
                    else { console.info('Task ' + config.task + ' started: ' + JSON.stringify(data.tasks))}
                    next(err);
                });
            }
        ], function (err) {
            if (err) {
                context.fail('An error has occurred: ' + err);
            }
            else {
                context.succeed('Successfully processed Amazon S3 URL.');
            }
        }
    );
};

The Lambda function uses the Async.js library to make it easier to program the sequence of events to perform in an event-driven language like Node.js. You can install the library by typing npm install async from within the directory where the Lambda function and its configuration file are located.

To upload the function into Lambda, zip the Lambda function, its configuration file, and the node_modules directory with the Async.js library. In the Lambda console, upload the .ZIP file as described in the Node.js for S3 events tutorial.

For this function to perform its job, it needs an IAM role with a policy that allows access to SQS as well as the right to start tasks on ECS. It also should be able to publish log data to CloudWatch Logs. However, it does not need explicit access to S3, because only the ECS task needs to download the source file from and upload the resulting image to S3. Here is an example policy:

{
    "Statement": [
        {
            "Action": [
                "logs:*", 
                "lambda:invokeFunction",
                "sqs:SendMessage",
                "ecs:RunTask"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:logs:*:*:*",
                "arn:aws:lambda:*:*:*:*",
                "arn:aws:sqs:*:*:*",
                "arn:aws:ecs:*:*:*"
            ]
        }
    ],
    "Version": "2012-10-17"
}

For the sake of simplicity in this post, we used very broadly defined resource identifiers like “arn:aws:sqs:*:*:*”, which cover all the resources of the given types. In a real-world scenario, we recommend that you make resource definitions as specific as possible by adding account IDs, queue names, and other resource ARN parameters.

Step 4: Configure S3 bucket notifications
Now you need to set up a bucket notification for your S3 bucket that triggers the Lambda function as soon as a new object is copied into the bucket.

This is a two-step process:

1. Add permission for S3 to be able to call your Lambda function:

$ aws lambda add-permission \
--function-name ecs-pov-ray-worker\
--region \
--statement-id \
--action "lambda:InvokeFunction" \
--principal s3.amazonaws.com \
--source-arn arn:aws:s3::: \
--source-account \
--profile

2. Set up an S3 bucket notification configuration (note: use the ARN for your Lambda function):

$ aws s3api put-bucket-notification-configuration \
--bucket \
--notification-configuration \
'{"LambdaFunctionConfigurations": [{"Events": ["s3:ObjectCreated:*"], "Id": "ECSPOVRayWorker", "LambdaFunctionArn": "arn:aws:lambda:eu-west-1::function:ecs-worker-launcher"}]}'

If you use these CLI commands, remember to substitute your particular name and Lambda function ARN.

Step 5: Create a Docker image
Docker images contain all of the software needed to run your application on Docker, out of Dockerfiles that describe the steps needed to create that image.

In this step, craft a Dockerfile that installs the POV-Ray ray-tracing application from POV (see important licensing information at the top of this post) as well as a simple shell script that uses the AWS CLI to consume messages from SQS, download and unpack the input data from S3 into the local file system, run the ray-tracer on the input file, then upload the resulting image back to S3, and delete the message from the SQS queue.

Start with the shell script, called ecs-worker.sh:

#!/bin/bash

# Copyright 2015 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License").
# You may not use this file except in compliance with the License.
# A copy of the License is located at
#
#     http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file.
# This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and limitations under the License.

#
# Simple POV-Ray worker shell script.
#
# Uses the AWS CLI utility to fetch a message from SQS, fetch a ZIP file from S3 that was specified in the message,
# render its contents with POV-Ray, then upload the resulting .png file to the same S3 bucket.
#

region=${AWS_REGION}
queue=${SQS_QUEUE_URL}

# Fetch messages and render them until the queue is drained.
while [ /bin/true ]; do
    # Fetch the next message and extract the S3 URL to fetch the POV-Ray source ZIP from.
    echo "Fetching messages from SQS queue: ${queue}..."
    result=$( \
        aws sqs receive-message \
            --queue-url ${queue} \
            --region ${region} \
            --wait-time-seconds 20 \
            --query Messages[0].[Body,ReceiptHandle] \
        | sed -e 's/^"\(.*\)"$/\1/'\
    )

    if [ -z "${result}" ]; then
        echo "No messages left in queue. Exiting."
        exit 0
    else
        echo "Message: ${result}."

        receipt_handle=$(echo ${result} | sed -e 's/^.*"\([^"]*\)"\s*\]$/\1/')
        echo "Receipt handle: ${receipt_handle}."

        bucket=$(echo ${result} | sed -e 's/^.*arn:aws:s3:::\([^\\]*\)\\".*$/\1/')
        echo "Bucket: ${bucket}."

        key=$(echo ${result} | sed -e 's/^.*\\"key\\":\s*\\"\([^\\]*\)\\".*$/\1/')
        echo "Key: ${key}."

        base=${key%.*}
        ext=${key##*.}

        if [ \
            -n "${result}" -a \
            -n "${receipt_handle}" -a \
            -n "${key}" -a \
            -n "${base}" -a \
            -n "${ext}" -a \
            "${ext}" = "zip" \
        ]; then
            mkdir -p work
            pushd work

            echo "Copying ${key} from S3 bucket ${bucket}..."
            aws s3 cp s3://${bucket}/${key} . --region ${region}

            echo "Unzipping ${key}..."
            unzip ${key}

            if [ -f ${base}.ini ]; then
                echo "Rendering POV-Ray scene ${base}..."
                if povray ${base}; then
                    if [ -f ${base}.png ]; then
                        echo "Copying result image ${base}.png to s3://${bucket}/${base}.png..."
                        aws s3 cp ${base}.png s3://${bucket}/${base}.png
                    else
                        echo "ERROR: POV-Ray source did not generate ${base}.png image."
                    fi
                else
                    echo "ERROR: POV-Ray source did not render successfully."
                fi
            else
                echo "ERROR: No ${base}.ini file found in POV-Ray source archive."
            fi

            echo "Cleaning up..."
            popd
            /bin/rm -rf work

            echo "Deleting message..."
            aws sqs delete-message \
                --queue-url ${queue} \
                --region ${region} \
                --receipt-handle "${receipt_handle}"

        else
            echo "ERROR: Could not extract S3 bucket and key from SQS message."
        fi
    fi
done

Remember to run chmod +x ecs-worker.sh to make the shell script executable. This permission is copied over to the Docker image you create in the next step, which is to put together a Dockerfile that includes all you need to set up the POV-Ray software and the AWS CLI in addition to your script:

# POV-Ray Amazon ECS Worker

FROM ubuntu:14.04

MAINTAINER FIRST_NAME LAST_NAME <EMAIL@DOMAIN.COM>

# Libraries and dependencies

RUN \
  apt-get update && apt-get -y install \
  autoconf \
  build-essential \
  git \
  libboost-thread-dev \
  libjpeg-dev \
  libopenexr-dev \
  libpng-dev \
  libtiff-dev \
  python \
  python-dev \
  python-distribute \
  python-pip \
  unzip \
  zlib1g-dev

# Compile and install POV-Ray

RUN \
  mkdir /src && \
  cd /src && \
  git clone https://github.com/POV-Ray/povray.git && \
  cd povray && \
  git checkout origin/3.7-stable && \
  cd unix && \
  sed 's/automake --w/automake --add-missing --w/g' -i prebuild.sh && \
  sed 's/dist-bzip2/dist-bzip2 subdir-objects/g' -i configure.ac && \
  ./prebuild.sh && \
  cd .. && \
  ./configure COMPILED_BY="FIRST_NAME LAST_NAME <EMAIL@DOMAIN.COM>" LIBS="-lboost_system -lboost_thread" && \
  make && \
  make install

# Install AWS CLI

RUN \
  pip install awscli

WORKDIR /

COPY ecs-worker.sh /

CMD [ "./ ecs-worker.sh" ]

Substitute your own name and email address into the COMPILED_BY parameter when using this Dockerfile. Also, this file assumes that you have the ecs-worker.sh script in your current directory when you create the Docker image.

After making sure the shell script is in the local directory and setting up the Dockerfile (and your account/credentials with Docker Hub), you can create the Docker image using the following commands:

$ docker build -t /: .
$ docker login -u -e
$ docker push

Step 6: Create an ECS task definition
Now that you have a Docker image ready to go, you can create an ECS task definition:

{
    "containerDefinitions": [
        {
            "name": "ECSPOVRayWorker",
            "image": "/:",
            "cpu": 512,
            "environment": [
                {
                    "name": "AWS_REGION",
                    "value": "<YOUR-CHOSEN-AWS-REGION>"
                },
                {
                    "name": "SQS_QUEUE_URL",
                    "value": "https://<YOUR_REGION>.queue.amazonaws.com/<YOUR_AWS_ACCOUNT_ID>/ECSPOVRayWorkerQueue"
                }
            ],
            "memory": 512,
            "essential": true
        }
    ],
    "family": "ECSPOVRayWorker"
}

When using this example task definition file, remember to substitute your own values for the Dockerhub user, repository, and tag as well as your chosen AWS region and SQS queue ARN.

Now, you’re ready to register your task definition with ECS:

$ aws ecs register-task-definition –family ECSPOVRayWorkerTask –cli-input-json file://task-definition-file.json

Remember, the ECS task family name “ECSPOVRayWorkerTask” corresponds to the “task” attribute of your Lambda function’s configuration file. This is how Lambda knows which ECS task to start upon invocation; if you decide to name your ECS task definition differently, also remember to update the Lambda function’s configuration file accordingly.

Step 7: Add a policy to your ECS instance role that allows access to SQS and S3
Your SQS queue worker script running inside your Docker container on ECS needs some permissions to fetch messages from SQS, download and upload files to/from S3 and to delete messages from SQS when it’s done.

The following example policy shows the permissions needed for this application, in addition to the standard ECS-related permissions:

{
    "Statement": [
        {
            "Action": [
                "s3:ListAllMyBuckets"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::*"
        },
        {
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::"
        },
        {
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::/*"
        }
    ],
    "Version": "2012-10-17"
}

You can attach this policy to your ECS instance role or work the missing policy statements into your existing ECS instance role. The former option is preferred as it lets you manage different policies for different tasks in separate policy documents.

Step 8: Test your new ray-tracing service
You’re now ready to test the new Lambda/Docker worker pattern for rendering ray-traced images!

To help you test it, we have provided you with a ready to use .ZIP file, containing a sample POV-Ray scene that you can download from the awslabs/lambda-ecs-worker-pattern GitHub repository.

Upload the .ZIP file to your S3 bucket and, after a few minutes, you should see the final rendered image appear in the same bucket, looking like this:

If it doesn’t work out of the box, don’t panic! Here are some hints on how to debug this scenario:

• Use the Amazon CloudWatch Logs console and look for errors generated by Lambda. Check out the Troubleshooting section of the AWS Lambda documentation.
• Log into your ECS container node(s) and use the docker ps -a command to identify the Docker container that was running your application. Use the docker logs command to look for errors. Check out the Troubleshooting section of the ECS documentation.
• If the Docker container is still running, you can log into it using the docker exec -it /bin/bash command and see what’s going on as it happens.

All in one go

To make setup even easier, we have put together a Python Fabric script that handles all of the above tasks for you. Fabric is a Python module that makes it easy to run commands on remote nodes, transfer files over SSH, run commands locally, and structure your script in a manner similar to a makefile.

You can download the script and Python fabfile that can help set this up, along with instructions, from the awslabs/lambda-ecs-worker-pattern repository on GitHub.

Further considerations

This example is intentionally simple and generic so you can adapt it to a wide variety of situations. When implementing this pattern for your own projects, you may want to consider additional issues.

In this pattern, each Lambda function launches its own ECS container to process the event. When many events occur, multiple containers are launched and this may not be what you want; a single running ECS task can continue processing messages from SQS until the queue is empty. Consider scaling the number of running ECS tasks independently of the number of Lambda function invocations. For more information, see Scaling Amazon ECS Services Automatically Using Amazon CloudWatch and AWS Lambda.

This approach is not limited to launching ECS containers; you can use Lambda to launch any other AWS service or resource, including Amazon Elastic Transcoder jobs, Amazon Simple Workflow Service executions, or AWS Data Pipeline jobs.

Combining ECS tasks with SQS is a very simple, but powerful batch worker pattern. You can use it even without Lambda: whenever you want to get a piece of long-running batch work done, write its parameters into an SQS queue and launch an ECS task in the background, while your application continues normally.

This pattern uses SQS to buffer the full S3 bucket notification event for the ECS task to pick it up. In cases where the parameters to be forwarded to ECS are short and simple (a URL, file name, or simple data structure), you can wrap them into environment variables and specify them as overrides in the run-task operation. This means that for simple parameters, you can stop using SQS altogether.

This pattern can also save on costs. Many traditional computing tasks need a specialized software installation (like the POV-Ray rendering software in this example) but are only used intermittently (such as one time per day or per week) for less than an hour. Keeping machines or fleets for such specialized tasks can create waste because they may not reach a significant use level.
Using ECS, you can share the hardware infrastructure for your batch worker fleets among very different specific worker implementations (a ray-tracing application, ETL process, document converter, etc.). This allows you to drive higher use of your generic ECS fleet while it performs very different tasks, through the ability to run different container images on the same infrastructure. You need fewer hardware resources to accommodate a wide variety of tasks.

Conclusion

This pattern is widely applicable. Many applications that can be seen as batch-driven workers can be implemented as ECS tasks that can be started from a Lambda function, using SQS to buffer parameters.

Look at your infrastructure and try to identify underused EC2 worker instances that could be re-implemented as ECS tasks; run them on a smaller, more efficient footprint and keep all of their functionality. Re-visit event-driven cases where you may have dismissed Lambda before, and try to apply the techniques outlined in this post in order to expand the usefulness of Lambda into more use cases with more complex execution requirements.

We hope you found this post useful and look forward to your comments about where you plan to implement this in your current and future projects.

Tipu Qureshi, AWS Senior Cloud Support Engineer

With Spot Instances, you can save up to 90% of costs by bidding on spare Amazon Elastic Compute Cloud (Amazon EC2) instances. This reference architecture is meant to help enable you to realize cost savings for batch processing applications while maintaining high availability. We recommend tailoring and testing it for your application before implementing it in a production environment.

Below is a multi-part job processing architecture that can be used for the deployment of a heterogeneous, scalable “grid” of worker nodes that can quickly crunch through large batch processing tasks in parallel. There are numerous batch oriented applications in place today that can leverage this style of on-demand processing, including claims processing, large scale transformation, media processing and multi-part data processing work.

Raw job data is uploaded to Amazon Simple Storage Service (S3) which is a highly-available and persistent data store. An AWS Lambda function will be invoked by S3 every time a new object is uploaded to the input S3 bucket. AWS Lambda is a compute service that runs your code in response to events and automatically manages the compute resources for you, making it easy to build applications that respond quickly to new information. Information about Lambda functions is available here and a walkthrough on triggering Lambda functions on S3 object uploads is available here.

AWS Lambda automatically runs your code with an IAM role that you select, making it easy to access other AWS resources, such as Amazon S3, Amazon SQS, and Amazon DynamoDB. AWS lambda can be used to place a job message into an Amazon SQS queue. Amazon Simple Queue Service (SQS) is a fast, reliable, scalable, fully managed message queuing service, which makes it simple and cost-effective to decouple the components of a cloud application. Depending on the application’s needs, multiple SQS queues might be required for different functions and priorities.

The AWS Lambda function will also store state information for each job task in Amazon DynamoDB. DynamoDB is a regional service, meaning that the data is automatically replicated across availability zones. AWS Lambda can be used to trigger other types of workflows as well, such as an Amazon Elastic Transcoder job. EC2 Spot can also be used with Amazon Elastic MapReduce (Amazon EMR).
Below is a sample IAM policy that can be attached to an IAM role for AWS Lambda. You will need to change the ARNs to match your resources.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1438283855455",
      "Action": [
        "dynamodb:PutItem"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:dynamodb:us-east-1::table/demojobtable"
    },
    {
      "Sid": "Stmt1438283929844",
      "Action": [
        "sqs:SendMessage"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:sqs:us-east-1::demojobqueue"
    }
  ]
}

Below is a sample Lambda function that will send an SQS message and put an item into a DynamoDB table in response to a S3 object upload. You will need to change the SQS queue URL and DynamoDB table name to match your resources.

// create an IAM Lambda role with access to Amazon SQS queue and DynamoDB table
// configure S3 to publish events as shown here: http://docs.aws.amazon.com/lambda/latest/dg/walkthrough-s3-events-adminuser-configure-s3.html

// dependencies
var AWS = require('aws-sdk');

// get reference to clients
var s3 = new AWS.S3();
var sqs = new AWS.SQS();
var dynamodb = new AWS.DynamoDB();

console.log ('Loading function');

exports.handler = function(event, context) {
  // Read options from the event.
  var srcBucket = event.Records[0].s3.bucket.name;
  // Object key may have spaces or unicode non-ASCII characters.
  var srcKey = decodeURIComponent(event.Records[0].s3.object.key.replace(/\+/g, " "));
  // prepare SQS message
  var params = {
    MessageBody: 'object '+ srcKey + ' ',
    QueueUrl: 'https://sqs.us-east-1.amazonaws.com//demojobqueue',
    DelaySeconds: 0
  };
  //send SQS message
  sqs.sendMessage(params, function (err, data) {
    if (err) {
      console.error('Unable to put object' + srcKey + ' into SQS queue due to an error: ' + err);
      context.fail(srcKey, 'Unable to send message to SQS');
    } // an error occurred
    else {
      //define DynamoDB table variables 
      var tableName = "demojobtable";
      var datetime = new Date().getTime().toString();
      //Put item into DynamoDB table where srcKey is the hash key and datetime is the range key
      dynamodb.putItem({
        "TableName": tableName,
        "Item": {
          "srcKey": {"S": srcKey },
          "datetime": {"S": datetime },
        }
      }, function(err, data) {
        if (err) {
          console.error('Unable to put object' + srcKey + ' into DynamoDB table due to an error: ' + err);
          context.fail(srcKey, 'Unable to put data to DynamoDB Table');
        }
        else {
          console.log('Successfully put object' + srcKey + ' into SQS queue and DynamoDB');
          context.succeed(srcKey, 'Data put into SQS and DynamoDB');
        }
      });
    }
  });
};

Worker nodes are Amazon EC2 Spot and On-demand instances on deployed Auto Scaling groups. These groups are containers that ensure health and scalability of worker nodes. Worker nodes pick up job parts from the input queue automatically and perform single tasks based on the job task state in DynamoDB. Worker nodes will store the input objects in a file system such as Amazon Elastic File System (Amazon EFS) for processing. Amazon EFS is a file storage service for EC2 that provides elastic capacity to your applications, automatically adding storage as you add files. Depending on IO and application needs, the job data can also be stored on local instance store or Amazon Elastic Block Store (EBS). Each job can be further split into multiples sub-parts if there is a mechanism to stitch the outputs together (as in the case of some media processing where pre-segmenting the output may be possible). Once completed, the objects will be uploaded back to S3 using multi-part upload.

Similar to our blog on an EC2 Spot architecture for web applications, Auto Scaling groups running On-demand instances can be used together with groups running Spot instances. Spot Auto Scaling groups can use different Spot bid prices and even different instance types to give you more flexibility and to meet changing traffic demands.

The availability of Spot instances can vary depending on how many unused Amazon EC2 instances are available. Because real-time supply and demand dictates the available supply of Spot instances, you should architect your application to be resilient to instance termination. When the Spot price exceeds the price you named (i.e. the bid price), the instance will receive a warning that it will be terminated after two minutes. You can manage this event by creating IAM roles with the relevant SQS and DynamoDB permissions for the Spot instances to run the required shutdown scripts. Details about Creating an IAM Role Using the AWS CLI can be found here and Spot Instance Termination Notices documentation can be found here.

A script like the following can be placed in a loop and can be run on startup (e.g. via systemd or rc.local) to detect for Spot instance termination. It can then update job task state in DynamoDB and re-insert the job task into the queue if required. We recommend that applications poll on the termination notice at five-second intervals.

#!/bin/bash
while true
  do
    if curl -s http://169.254.169.254/latest/meta-data/spot/termination-time | grep -q .*T.*Z; then /env/bin/runterminationscripts.sh; 
  else
    # Spot instance not yet marked for termination.
    sleep 5
  fi
done

An Auto Scaling group running on-demand instances in tandem with Auto Scaling group(s) running Spot instances will help to ensure your application’s availability in case of changes in Spot market price and Spot instance capacity. Additionally, using multiple Spot Auto Scaling groups with different bids and capacity pools (groups of instances that share the same attributes) can help with both availability and cost savings. By having the ability to run across multiple pools, you reduce your application’s sensitivity to price spikes that affect a pool or two (in general, there is very little correlation between prices in different capacity pools). For example, if you run in five different pools your price swings and interruptions can be cut by 80%.

For Auto Scaling to scale according to your application’s needs, you must define how you want to scale in response to changing conditions. You can assign more aggressive scaling policies to Auto Scaling groups that run Spot instances, and more conservative ones to Auto Scaling groups that run on-demand instances. The Auto Scaling instances will scale up based on the SQS queue depth CloudWatch metric (or more relevant metric depending on your application), but will scale down based on EC2 instance CPU utilization CloudWatch metric (or more relevant metric) to ensure that a job actually gets completed before the instance is terminated. For information about using Amazon CloudWatch metrics (such as SQS queue depth) to scale automatically, see Dynamic Scaling. As a safety buffer, a grace period (e.g. 300 seconds) should also be configured for the Auto Scaling group to prevent termination before a job completes.

Reserved instances can be purchased for the On-demand EC2 instances if they are consistently being used to realize even more cost savings.

To automate further, you can optionally create a Lambda function to dynamically manage Auto Scaling groups based on the Spot market. The Lambda function could periodically invoke the EC2 Spot APIs to assess market prices and availability and respond by creating new Auto Scaling launch configurations and groups automatically. Information on the Describe Spot Price History API is available here. This function could also delete any Spot Auto Scaling groups and launch configurations that have no instances. AWS Data Pipeline can be used to invoke the Lambda function using the AWS CLI at regular intervals by scheduling pipelines. More information about scheduling pipelines is available here and information about invoking AWS Lambda functions using AWS CLI is here.

Tim Wagner, AWS Lambda General Manager

SQL databases have offered triggers for years, making it easy to validate and check data, maintain integrity constraints, create compute columns, and more. Why should SQL tables have all the fun…let’s do the equivalent for NoSQL data!

Amazon DynamoDB recently launched their streams feature (table update notifications) in production. When you combine this with AWS Lambda, it’s easy to create NoSQL database triggers that let you audit, aggregate, verify, and transform data. In this blog post we’ll do just that: first, we’ll create a data audit trigger, then we’ll extend it to also transform the data by adding a computed column to our table that the trigger maintains automatically. We’ll use social security numbers and customer names as our sample data, because they’re representative of something you might find in a production environment. Let’s get started…

Data Auditing

Our first goal is to identify and report invalid social security numbers. We’ll accept two formats: 9 digits (e.g., 123456789) and 11 digits (e.g., 123-45-6789). Anything else is an error and will generate an SNS message which, for the purposes of this demo, we’ll use to send an email report of the problem.

Setup Part 1: Defining a Table Schema

First, start by creating a new table; I’m calling it “TriggerDemo”:

For our example we’ll use just two fields: Name and SocialSecurityNumber (the primary hash key and primary range key, respectively, both represented as strings). In a more realistic setting you’d typically have additional customer-specific information keyed off these fields. You can accept the default capacity settings and you don’t need any secondary indices.

You do need to turn on streams in order to be able to send updates to your AWS Lambda function (we’ll get to that in a minute). You can read more about configuring and using DynamoDB streams in the DynamoDB developer guide.

Here’s the summary view of the table we’ve just configured:

Setup Part 2: SNS Topic and Email Subscription

To give us a way to report errors, we’ll create an SNS topic; I’m calling mine, “BadSSNNumbers”.

(The other topic here is the DynamoDB alarm.)

…and then I’ll subscribe my email to it to receive error notifications:

(I haven’t shown it here, but you can also turn on SNS logging as a debugging aid.)

Ok, we have a database and a notification system…now we need a compute service!

Setup Part 3: A Lambda-based Trigger

Now we’ll create an AWS Lambda function that will respond to DynamoDB updates by verifying the integrity of each social security number, using the SNS topic we just created to notify us of any problematic entries.

First, create a new Lambda function by selecting the “dynamodb-process-stream” blueprint. Blueprints help you get started quickly with common tasks.

For the event source, select your TriggerDemo table:

You’ll also need to provide your function with permissions to read from the stream by choosing the recommended role (DynamoDB event stream role):

The blueprint-provided permission policy only assumes you’re going to read from the update stream and create log entries, but we need an additional permission: publishing to the SNS topic. In the later part of this demo we’ll also want to write to the table, so let’s take care of both pieces at once: Hop over to the IAM console and add two managed policies to your role: SNS full access and DynamoDB full access. (Note: This is an quick approach for demo purposes, but if you want to use the techniques described here for a production table, I strongly recommend your create custom “minimal trust” policies that permit only the necessarily operations and resources to be accessed from your Lambda function.)

The code for the Lambda function is straightforward: It receives batches of change notifications from DynamoDB and processes each one in turn by checking its social security number, reporting any malformed ones via the SNS topic we configured earlier. Replace the sample code provided by the blueprint with the following, being sure to replace the SNS ARN with the one from your own topic:

var AWS = require('aws-sdk');
var sns = new AWS.SNS();
exports.handler = function(event, context) {processRecord(context, 0, event.Records);}

// Process each DynamoDB record
function processRecord(context, index, records) {
    if (index == records.length) {
        context.succeed("Processed " + records.length + " records.");
        return;
    }
    record = records[index];
    console.log("ID: " + record.eventID + "; Event: " + record.eventName);
    console.log('DynamoDB Record: %j', record.dynamodb);
    // Assumes SSN# is only set only on row creation
    if ((record.eventName != "INSERT") || valid(record)) processRecord(context, index+1, records);
    else {
        console.log('Invalid SSN # detected');
        var name = record.dynamodb.Keys.Name.S;
        console.log('name: ' + name);
        var ssn  = record.dynamodb.Keys.SocialSecurityNumber.S;
        console.log('ssn: ' + ssn);
        var message = 'Invalid SSN# Detected: Customer ' + name + ' had SSN field of ' + ssn + '.';
        console.log('Message to send: ' + message);
        var params = {
            Message:  message,
            TopicArn: 'YOUR BadSSNNumbers SNS ARN GOES HERE'
        };
        sns.publish(params, function(err, data) {
            if (err) console.log(err, err.stack);
            else console.log('malformed SSN message sent successfully');
            processRecord(context, index+1, records);
        });
    }
}

// Social security numbers must be in one of two forms: nnn-nn-nnnn or nnnnnnnnn.
function valid(record) {
    var SSN = record.dynamodb.Keys.SocialSecurityNumber.S;
    if (SSN.length != 9 && SSN.length != 11) return false;
    if (SSN.length == 9) {
        for (var indx in SSN) if (!isDigit(SSN[indx])) return false;
        return true;
    }
    else {
        return isDigit(SSN[0]) && isDigit(SSN[1]) && isDigit(SSN[2]) &&
               SSN[3] == '-' &&
               isDigit(SSN[4]) && isDigit(SSN[5]) &&
               SSN[6] == '-' &&
               isDigit(SSN[7]) && isDigit(SSN[8]) && isDigit(SSN[9]) && isDigit(SSN[10]);
    }
}

function isDigit(c) {return c >= '0' && c <= '9';}

Testing the Trigger

Ok, now it’s time to see things in action. First, use the “Test” button on the Lambda console to validate your code and make sure the SNS notifications are sending email. Next, if you created your Lambda function event source in a disabled state, enable it now. Then go to the DynamoDB console and enter some sample data. First, let’s try a valid entry:

Since this one was valid, you should get a CloudWatch Log entry but no email. Now for the fun part: Try an invalid entry, such as “Bob Smith” with a social security number of “asdf”. You should receive an email notification something like this for the invalid SSN entry:

You can also check the Amazon CloudWatch Logs to see the analysis and reporting in action and debug any problems:

So in a few lines of Lambda function code we implemented a scalable, serverless NoSQL trigger capable of auditing every change to a DynamoDB table and reporting any errors it detects. You can use similar techniques to validate other data types, aggregate or mark suspected errors instead of reporting them via SNS, and so forth.

Data Transformation

In the previous section we audited the data. Now we’re going to take it a step further and have the trigger also maintain a computed column that describes the format of the social security number. The computed attribute can have one of three values: 9 (meaning, “The social security number in this row is valid and is a 9-digit format”), 11, or “INVALID”.

We don’t need to alter anything about the DynamoDB table or the SNS topic, but in addition to the extra code, the IAM permissions for the Lambda function must now allow us to write to the DynamoDB table in addition to reading from its update stream. If you added the DynamoDBFullAccess managed policy earlier when you did the SNS policy, you’re already good. If not, hop over to the IAM console and add that second managed policy now. (Also see the best practice note above on policy scoping if you’re putting this into production.)

The code changes only slightly to add the new DynamoDB writes:

var AWS = require('aws-sdk');
var sns = new AWS.SNS();
var dynamodb = new AWS.DynamoDB();
exports.handler = function(event, context) {processRecord(context, 0, event.Records);}

function processRecord(context, index, records) {
    if (index == records.length) {
        context.succeed("Processed " + records.length + " records.");
        return;
    }
    record = records[index];
    console.log("ID: " + record.eventID + "; Event: " + record.eventName);
    console.log('DynamoDB Record: %j', record.dynamodb);
    if (record.eventName != "INSERT") processRecord(context, index+1, records);
    else if (valid(record)) {
        var name = record.dynamodb.Keys.Name.S;
        var ssn = record.dynamodb.Keys.SocialSecurityNumber.S;
        dynamodb.putItem({
            "TableName":"TriggerDemo",
            "Item": {
                "Name":                 {"S": name},
                "SocialSecurityNumber": {"S": ssn},
                "SSN Format":           {"S": ssn.length == 9 ? "9" : "11"}
            }
        }, function(err, data){
            if (err) console.log(err, err.stack);
            processRecord(context, index+1, records);
        });
    }
    else {
        console.log('Invalid SSN # detected');
        var name = record.dynamodb.Keys.Name.S;
        console.log('name: ' + name);
        var ssn  = record.dynamodb.Keys.SocialSecurityNumber.S;
        console.log('ssn: ' + ssn);
        var message = 'Invalid SSN# Detected: Customer ' + name + ' had SSN field of ' + ssn + '.';
        console.log('Message to send: ' + message);
        var params = {
            Message:  message,
            TopicArn: 'YOUR BadSSNNumbers SNS ARN GOES HERE'
        };
        sns.publish(params, function(err, data) {
            if (err) console.log(err, err.stack);
            else console.log('malformed SSN message sent successfully');
            dynamodb.putItem({
                "TableName":"TriggerDemo",
                "Item": {
                    "Name":                 {"S": name},
                    "SocialSecurityNumber": {"S": ssn},
                    "SSN Format":           {"S": "INVALID"}
                }
            }, function(err, data){
                if (err) console.log(err, err.stack);
                processRecord(context, index+1, records);
            });
        });
    }
}

// Social security numbers must be in one of two forms: nnn-nn-nnnn or nnnnnnnnn.
function valid(record) {
    var SSN = record.dynamodb.Keys.SocialSecurityNumber.S;
    if (SSN.length != 9 && SSN.length != 11) return false;
    if (SSN.length == 9) {
        for (var indx in SSN) if (!isDigit(SSN[indx])) return false;
        return true;
    }
    else {
        return isDigit(SSN[0]) && isDigit(SSN[1]) && isDigit(SSN[2]) &&
               SSN[3] == '-' &&
               isDigit(SSN[4]) && isDigit(SSN[5]) &&
               SSN[6] == '-' &&
               isDigit(SSN[7]) && isDigit(SSN[8]) && isDigit(SSN[9]) && isDigit(SSN[10]);
    }
}

function isDigit(c) {return c >= '0' && c <= '9';}

Now you can go to the DynamoDB console to add more rows to your table to watch your trigger both check and your entries and maintain a computed format column for them. Don’t forget to refresh the DynamoDB table browse view to see the updates!

(Now that the code updates rows in the original table, testing from the Lambda console will generate double notifications – the first one from the original test, and the second when the item is created for real. You could add an “istest” field to the sample event in the console test experience and a condition in the code to prevent this if you want to keep testing “offline” from the actual table.)

I chose to leave the original data unchanged in this example, but you could also use the trigger to transform the original values instead – for example, choosing the 11-digit format as the canonical one and then converting any 9-digit values into their 11-digit equivalents.

Summary

In this post we explored combining DynamoDB stream notifications with AWS Lambda functions to recreate conventional database triggers in a serverless, NoSQL architecture. We used a simple nodejs function to first audit and later transform rows in the table in order to find invalid social security numbers and to compute the format of the number in each entry. While we worked in JavaScript for this example, you could also use Java, Clojure, Scala, or other jvm-based languages to write your trigger. Our notification method of choice for the demo was an SNS-provided email, but text messages, web hooks, and SQS entries just require a different subscription.

Until next time, happy Lambda (and database trigger) coding!

-Tim

 
Follow Tim’s Lambda adventures on Twitter

Tim Wagner, AWS Lambda General Manager

Today Amazon S3 added some great new features for event handling:

• Prefix filters – Send events only for objects in a given path
• Suffix filters – Send events only for certain types of objects (.png, for example)
• Deletion events

You can see some images of the S3 console’s experience on the AWS Blog; here’s what it looks like in Lambda’s console: .

Let’s take a look at how these new features apply to Lambda event processing for S3 objects.

Processing Deletions

Previously, you could get S3 bucket notification events (aka “S3 triggers”) when objects were created but not when they were deleted. With the new event type, you can now use Lambda to automatically apply cleanup code when an object goes away, or to help keep metadata or indices up to date as S3 objects come and go.

You can request notification when an object is deleted by using the s3:ObjectRemoved:Delete event type. You can request notification when a delete marker is created for a versioned object by using s3:ObjectRemoved:DeleteMarkerCreated. You can also use a wildcard expression like s3:ObjectRemoved:* to request notification any time an object is deleted, regardless of whether it’s been versioned.

S3 events aren’t guaranteed to be sent in order, but S3 does offers help for keeping the event sequence organized: The S3 event structure includes a new field, sequencer, which is a hexadecimal value that establishes a relative ordering among PUTs and DELETEs for the same object. (The values can’t be meaningfully compared across different objects.) If you’re trying to keep another data structure, like an index, in sync this is critical information to save and compare, as a PUT followed by a DELETE is very different from a DELETE followed by a PUT. Check out the S3 event format for a detailed description of the event format and options.

Prefix Filters

Prefix filters can be used to pick the “directory” in which to send events. For example, let’s say you have two paths in use in your S3 bucket:

Incoming/
Thumbnails/

Your client app uploads images to the Incoming path and, for each image, you create a matching thumbnail in the Thumbnails path.

Previously, to use Lambda for this you’d need to send all events from the bucket to Lambda – including the thumbnail creation events. That has a couple of downsides: You have to send more events than you’d like, and you have to explicitly code your Lambda function to ignore the “extra” ones in order to avoid recursively creating thumbnails of thumbnails (of thumbnails of …)

Now, it’s easier: You set a prefix filter of “Incoming/”, and only the incoming images are sent to Lambda. The Lambda code also gets simpler because the recursion check is no longer needed, since you never write images to the path in your prefix filter. (Of course it doesn’t hurt to retain it for safety if you prefer.)

You can also use prefix filters on the object name (the “path” is just a string in reality). So if you name files like, “IMAGE001″ and “DOC002″ and you only want to send documents to Lambda, you can set a prefix of “DOC”.

Suffix Filters

Suffix filters work similarly, although they’re conventionally used to pick the “type” of file to send to Lambda by choosing a suffix, such as “.doc” or “.java”. You can combine prefix and suffix filters.

In general, you can’t have define overlapping prefix or suffix filters, as it would make the delivery ambiguous; the partitioning of events must be unique. The exception to this is that you can have overlapping prefix filters if the suffix filters disambiguate. If you want to send one event to multiple recipients, check out the post on S3 event fanout to see some suggested architectures for doing just that.

You can read all the details on Amazon S3 event notification settings in the docs, which also cover using the command line and programmatic/REST access.

Until next time, happy Lambda (and S3 event) coding!

-Tim

 
Follow Tim’s Lambda adventures on Twitter

John Stamper, AWS Solution Architect

Use Cases

The above architecture is an event-driven general-purpose parallel data processing system – data enters S3, notification of new data is sent to SNS, which packages the S3 event notification as a message and delivers it to subscribers. This architecture is ideal for workloads that need more than one data derivative of an object. The purpose of the subscribers is to create a layer of processing which accommodates a wide variety of data sizes and subsequently send the results of processing to some storage layer. The architecture is not prescriptive with regard to the post-processing storage layer and is out of scope for this article. In the illustration above, black arrows depict data and blue arrows depict event notifications.

Example use cases are described below.

Image Processing
Master image must be processed to produce multiple image derivatives, e.g. resized, OpenCV result.

Application Log Processing
Application log data must be processed to produce multiple log derivatives, e.g. formatted for operations, security, marketing.

Content Transformation
Documents of one format, e.g. Microsoft Word, must be converted to multiple other formats such as PDF, RTF, MHTML, and ODT.

SNS supports message delivery to several types of subscribers, notably Lambda functions, SQS queues, and HTTP/HTTPS endpoints. Lambda functions make it easy to respond to data without the need for servers. To process data with a long-running or existing application, you can also use SNS to easily send the message to an SQS queue or HTTP/HTTPS endpoint. At this stage messages can be processed by EC2, which offers a wide range of compute/memory/storage options. Overall, the architecture can provide an event-driven parallel data processing system that can leverage the entire AWS compute offering.

This article will focus on the steps to configure a S3 bucket to send an event notification to a SNS topic and subscribe two Lambda functions to the topic. The resulting architecture is a simple implementation of S3 event notification fanout to Lambda functions for processing, which is applicable for workloads that require multiple data derivatives of the same object. In the center of the architecture is the ‘event manifold’, similar to a mechanical manifold, which intakes an event notification at one end (S3), transforms it to a message, and distributes it to all subscribers (data processing elements). This architecture allows customers to build an event-driven parallel data processing architecture that is fast, flexible, and easy to maintain over time. Below is an illustration of the architecture to be assembled.

Step 1 – Create the Bucket

To create a Bucket, follow the documentation here. For this article, set the bucket name to event-manifold-bucket.

Step 2 – Create the Topic

To create a Topic, follow the documentation here. For this article, set the topic name to event-manifold-topic.

Step 3 – Update the Topic Policy to allow Event Notifications from an S3 Bucket

The Topic’s Policy must permit an S3 Bucket to publish event notifications to it. To do so, do the following:

1. Select the event-manifold-topic
2. Select Edit Topic Policy from the Actions button



3. Select the Advanced Tab
4. Clear the existing Policy and replace it with the following policy statement:
   
   • Replace ‘region’ with the region in which the Topic is located, e.g. us-west-1.
   • Replace ‘account id’ with the account id of the Topic, e.g. 123456789012.
   • Replace ‘topic name’ with event-manifold-topic.
   • Replace ‘bucket name’ with event-manifold-bucket.
5. Select the Update Policy button

At this point we have the S3 bucket, a SNS Topic, and the Topic is configured to permit our specific bucket to call the Publish API on it. Next we will configure the S3 bucket to send notifications to the Topic. We will choose the event type ObjectCreated (All) for our general purpose data processing architecture.

Step 4 – Configure the S3 Bucket to send Event Notifications to the SNS Topic

1. Select the ‘Events’ portion of the S3 bucket created in Step 1.
2. Enter a name for the notification, e.g. s3fanout
3. Enter an event type for the notification, e.g. ObjectCreated (All)
4. Select SNS Topic radio button of the Send To radio button group
5. Select Add SNS topic ARN from the SNS Topic drop down list
6. Enter the SNS Topic ARN created in Step 2
7. Click the Save button

The picture below is an example.

Step 5 – Create the IAM Role for the Lambda functions

In this step you will create an IAM role which grants permissions for the Lambda functions to write to CloudWatch Logs and read objects from the originating S3 bucket.

1. In the IAM portion of the AWS console, click the Policies link on the left.
2. Select the Create Policy button at the top.
3. In the Create Policy window, select the button which corresponds to the Policy Generator
4. In the Permissions window, add two statements
   1. Statement One
      1. Effect = Allow
      2. AWS Service = Amazon CloudWatch Logs
      3. Actions = CreateLogGroup, CreateLogStream, PullLogEvents
      4. ARN = arn:aws:logs:*:*:*
      5. Select the Add Statement button
   2. Statement Two
      1. Effect = Allow
      2. AWS Service = Amazon S3
      3. Actions = GetObject
      4. ARN = arn:aws:s3:::event-manifold-bucket
      5. Select the Add Statement button
5. Select the Next Step button.
6. In the Review Policy window, enter a name for the Policy, e.g. ‘Fanout-Lambda-Policy’.
7. Select the Create Policy button.
8. In the IAM portion of the AWS console, click the Roles link on the left.
9. Click the Create New Role button at the top.
10. In the Select Role Name window, enter a name for the Role, e.g. CloudWatchLogs-Write-S3Bucket-Read. Click the Next Step button.
11. In the Select Role Type window, select the button which corresponds to AWS Lambda.
12. In the Attach Policy window, select the policy you previously created, ‘Fanout-Lambda-Policy’.
13. In the Review Policy window, select the Create Role button.

Step 6 – Create the Lambda functions

In this step you will create two Lambda functions in the same region as the SNS Topic, data-processor-1 and data-processor-2. Each function will be edited inline and their execution role will be the role CloudWatchLogs-Write-S3Bucket-Read created in Step 5. This role provides visibility into what the functions are doing through simple logging statements and allows the functions to read the objects from S3.

1. In the Lambda portion of the AWS console, click the Create a Lambda Function button.
2. On the Select Blueprint page, select the sns-message blueprint option.
3. On the Configure event sources page, select event-manifold-topic from the SNS topic dropdown list. Click Next.
4. On the Configure Function page, enter a name for the function, e.g. ‘data-processor-1.
5. As an option, enter a description of the function in the Description field.
6. Use the default Runtime, Node.js.
7. Use the default Code entry type, Edit code inline.
8. Use the default Handler, index.handler.
9. Select the CloudWatchLogs-Write-S3Bucket-Read role from the Role drop down list.
10. Click the Next button.
11. On the Review page, select the Enable now radio button.
12. Click the Create Lambda Function button at the bottom of the page.

Repeat steps 1-12 to create a second Lambda function, setting the name to data-processor-2 in step 4.

At this point you have two Lambda functions, each of which programmed to receive an event notification record from a SNS Topic and log to CloudWatch Logs the incoming event data and the SNS message portion of the record. The picture below shows the code.

Step 7 – Modify the Lambda functions to process SNS messages of S3 event notifications

The code to process a SNS message delivery is shown above. A SNS message delivery is a JSON object containing an array named ‘Records’ with one element within the array – the SNS Message Delivery Object. The element contains several items of data about the event. An example of the SNS message delivery to a Lambda function is below.

For the S3 event notification fanout architecture (S3 publish event notification -> SNS Topic -> SNS message delivery of S3 publish event notification -> Lambda function), the JSON object received by the Lambda function is different from a JSON object from a SNS message delivery. The S3 event notification is contained within the Sns.Message attribute of the SNS Message Delivery Object. An example of a SNS message delivery of a S3 event notification is shown below.

Some extra code is needed for the Lambda function to process the object created in S3. First the code must capture the Sns.Message object from the incoming record. Next, that object must be processed to unbundle the JavaScript object of the S3 event notification from the Sns.Message attribute. Example Lambda code to do this is shown below.

exports.handler = function(event,context) {
   var snsMsgString = JSON.stringify(event.Records[0].Sns.Message);
   var snsMsgObject = getSNSMessageObject(snsMsgString);
   var srcBucket = snsMsgObject.Records[0].s3.bucket.name;
   var srcKey = snsMsgObject.Records[0].s3.key;
   console.log(‘SRC Bucket: ’ + srcBucket);
   console.log(‘SRC Key: ’ + srcKey);
   …

The function getSNSMessageObject(string) must be included in your Lambda function and is shown below.

function getSNSMessageObject(msgString) {
   var x = msgString.replace(/\\/g,’’);
   var y = x.substring(1,x.length-1);
   var z = JSON.parse(y);
   
   return z;
}

Below is the Lambda function for data-processor-1.

Test the architecture

The architecture illustrated in the beginning has been created and assembled. When new objects are created in the event-manifold-bucket, S3 will send the event notification to the SNS Topic event-manifold-topic, which will subsequently deliver a message to both Lambda functions of the new object creation. You can see this by inspecting the logs in Amazon CloudWatch.

By selecting the Log Group for each Lambda function, you can see that both functions received the notification of the S3 create object event and they each have the data they need to pull the record from S3 and process it.

As an alternative to viewing the log output of the Lambda functions in CloudWatch, you can also view metrics in CloudWatch provided by SNS Topics, including NumberOfMessagesPublished, PublishSize, NumberOfNotificationsDelivered, and NumberOfNotificationsFailed.

Alternative Architecture

The architecture described in this article is one option available to customers who require an event-driven general parallel data-processing system. Another option to achieve S3 Fanout of Event Notifications is to configure a S3 bucket to send the event notification directly to a ‘master’ Lambda function. In this approach, the ‘master’ Lambda function replaces the SNS topic (the event manifold) and must be programmed to send data to the various elements of the processing layer.

Leveraging a Lambda function to serve as the event manifold provides the architect a high degree of choice with regard to the processing elements due to the flexibility offered by the current runtime environments of Lambda, Node.js and Java8. In addition, processing elements do not need to ‘unbundle’ the S3 event notification from the SNS.Message attribute of the Message Delivery Object. In exchange for high choice and reduced software maintenance, the architect receives additional maintenance of the ‘master’ Lambda function – one unit for every downstream data processing element. Below is an illustration of the alternative architecture.

Fast, Flexible, Easy to Maintain

The speed of the system is optimal since notifications of S3 events are event-driven and are delivered in parallel to subscribers, which results in parallel processing of data.

The flexibility of the system is high – to date SNS supports two endpoints that are capable of runtime processing: Lambda and HTTP/HTTPS endpoints, and SQS facilitates processing by queue consumers. Additional processing subscribers can be easily added or removed per business requirements.

The subscriber processing layer elements and the post-processing storage layer drives the maintenance of the system. The ingest storage layer (S3) is a highly scalable, reliable, and low-latency data storage infrastructure and the event manifold component (SNS) is a highly scalable, flexible, cost-effective notification service which require no ongoing maintenance.

Conclusion

The above architectures describe and illustrate event-driven general purpose parallel data processing systems. The first architecture utilizes at least three AWS services: Amazon S3, Amazon SNS, and AWS Lambda, with SNS serving as the ‘event manifold’. The alternative architecture utilizes at least two AWS services: Amazon S3 and AWS Lambda, with a Lambda function serving as the ‘event manifold’. These architectures are designed to support Internet scale data processing workloads, require low operational maintenance, provide the architect the option of leveraging the entire AWS Compute family for processing, and are flexible to dynamic business requirements for derivatives of data objects.

Tim Wagner, AWS Lambda

Great post on the AWS Big Data Blog by Martin Holste, a co-founder of the Threat Analytics Platform at FireEye, on using AWS Lambda to create scalable applications without infrastructure:

Building Scalable and Responsive Big Data Interfaces with AWS Lambda.

-Tim
Follow my Lambda adventures on Twitter

Tim Wagner, AWS Lambda General Manager


Will Gaul, AWS Lambda Software Developer

With the recent release of Amazon API Gateway, developers can now create custom RESTful APIs that trigger AWS Lambda functions, allowing for truly serverless backends that include built-in authorization, traffic management, monitoring, and analytics. To showcase what’s possible with this new integration and just how easy it is to build a service that runs entirely without servers, we’ve built SquirrelBin, a simple website that allows users to CRUD runnable little nuggets of code called acorns. Let’s take a look at how we made SquirrelBin…

The SquirrelBin Architecture

The following diagram illustrates SquirrelBin’s architecture:

Notice what isn’t in the diagram above: servers. In fact, no infrastructure is required for any part of the experience.

There is a clean separation between data management and presentation. The website itself is a fully client-side single page app written in Angular and hosted statically on Amazon S3, with DNS managed by Amazon Route 53. To manage acorns, the app makes REST calls to Amazon API Gateway. These endpoints then trigger Lambda functions that either interact with SquirrelBin’s underlying Amazon DynamoDB acorn-store or run the actual acorn code. Because SquirrelBin’s API is on a completely separate Lambda-powered stack it would be easy to create additional clients, such as an iOS app that calls the Lambda functions via the AWS Mobile SDK, or an Alexa Skill that enables you to run acorns with your voice.

You might have noticed that the Lambda control plane consists of five functions, one for each CRUD operation. All of these functions are just instances of the new microservice-http-endpoint blueprint available now in the Lambda console:

Yep, we wrote no code for any of the CRUD operations for the site!

While identical, each function is each only responsible for handling requests for its respective endpoint. This architecture presents a number of advantages:

• Isolated deployments and failures: A problem with your API no longer takes down your entire backend. Each Lambda function operates individually and can be edited without affecting other functions.
• Per-endpoint analytics for free: Each function will publish metrics on request count, errors, and more to CloudWatch, enabling you to quickly answer questions like, “How many acorns have been created in the last 24 hours?”
• Modularity, simplicity, and separation of concerns: Because each function is only responsible for doing One Thing Well ^TM, it becomes easier to manage end-to-end configuration, code logic, and service integrations.

That covers the CRUD operations. Executing an acorn is just as easy: A simple version of code execution that supports nodejs can be written in just four lines of code:

exports.handler = function(event, context) {
    if (event.language === 'javascript') context.succeed(eval(event.code.replace(/require/g, '')));
    else context.fail('Language not supported');
};

Developing SquirrelBin

We began the development process with an understanding of our desired architecture and set up each component within minutes, all from within the AWS console. (In fact, the longest part of the backend setup was waiting for the DNS record to propagate!)

The website itself was developed in tandem, at first using hard-coded mock data and Angular’s $timeout service to simulate REST calls. Once the basic page layouts were complete we swapped in the API Gateway URL and began interacting with live data. In total, SquirrelBin runs in about 150 lines of client-side JavaScript. You can see the full source self-hosted on SquirrelBin here.

We hope that this post has illustrated the simplicity and power of serverless backends made possible with Amazon API Gateway and AWS Lambda, and has inspired you to try making your own!

Until next time, happy Lambda (and SquirrelBin) coding!

-Will and Tim

 
Follow Tim’s Lambda adventures on Twitter

Tim Wagner, AWS Lambda


The 2015 AWS NY summit a lot of exciting content for AWS Lambda and ECS. If you weren’t able to join us there, here’s a summary of slideshares and videos with compute-related announcements and content:

Werner Vogels Keynote: Announcing Amazon API Gateway

Announcement excerpt:

Full keynote:

Werner and Matt Wood announce the new Amazon API Gateway and its integration with AWS Lambda.
 

Breakout session: AWS Lambda, Event-driven Code in the Cloud

See the Slides
Tim’s talk on AWS Lambda, with announcements about HTTP endpoint support, new features, and a sneak peak at the upcoming release of versioning. Also a fun guest appearance by Ricky Robinett of Twilio on integration with Lambda.

Breakout session: Build and Manage Your APIs with Amazon API Gateway

See the Slides
Simon Poile’s deep dive on the new Amazon API Gateway and its integration with AWS Lambda.
 

Breakout session: Amazon EC2 Container Service: Manage Docker-Enabled Apps in Amazon EC2

See the Slides
Brandon Chavis, AWS Solutions Architect, discusses how Amazon ECS makes it easy to build and deploy Docker-based applications.
 

Breakout session: Build Your Mobile App Faster with AWS Mobile Services

See the Slides
John Burry, AWS Principal Solutions Architect, discusses how you can quickly deliver mobile solutions with scalable backends using AWS services such as Amazon Cognito, Amazon API Gateway, and AWS Lambda.
 

-Tim
Follow my Lambda adventures on Twitter

Meta-announcement: We’re in the process of enabling commenting support for the AWS compute blog. You’ll see it starting to appear on newer posts and getting phased in for a subset of older ones. Looking forward to engaging with our readers directly!

-Tim, Deepak, and our many guest authors