This is a guest post by Çağatay Gürtürk, the creator of the Lambada framework

Serverless computing has become a hot topics since AWS Lambda and Amazon API Gateway started to offer an elegant way to build and deploy REST APIs without needing to maintain 24/7 running servers and infrastructure, with attractive pricing models.

Being the first language offered by Lambda, Node.JS seems to have the most online resources and tools but it is also possible to write Lambda functions natively with Java and Python. Java is especially interesting as a language because of its maturity, large community, and available codebase. With Lambda and Java, it is even possible to apply enterprise patterns and frameworks such as Spring, as well as all the best practices we used to apply in the Java world.

In order to make development for Lambda in Java easier, I started Lambada Framework as an open source project. It is a little, but powerful, open source project in beta stage that lets developers create a new serverless API in AWS infrastructure or migrate an existing one.

Lambada Framework accomplishes this target by implementing the most common JAX-RS annotations and providing a Maven plugin to deploy easily to the AWS cloud. Briefly, JAX-RS is a standard annotation set which can be used to map regular Java methods to HTTP paths and methods. For instance, you can look at the following method:

@GET
@Path("/helloworld/{id}")
public Response indexEndpoint(@PathParam int id) {
    return Response.status(200).entity("Hello world: " + id).build();
}

This is a very lean method marked with @GET and @Path annotations, which mean that this method is called when a GET request comes to URLs in "/helloworld/{id}" format, with theid parameter as an argument. Finally, it returns a Response object within this method with a 200 response code and text content. As you can see, these annotations offer a seamless way to define a REST API and map different resources to Java methods.

JAX-RS annotations on their own do not mean so much and they do not have any effect out-of-the-box. To make these annotations work, a JAX-RS implementation framework should be added to the project. This framework would scan all the JAX-RS annotations in the project and create a server and routing table to respond to HTTP requests correctly. While Jersey is one such reference implementation, and the most popular one, there are also other implementations of JAX-RS, such as RESTEasy and Apache CXF. You are free to choose any of them and your controller methods always stay same, thanks to standard annotations.

Lambada Framework is a JAX-RS implementation but different from the others: instead of running a web server, it scans the JAX-RS annotations at build time and populates Lambda functions and the API Gateway definitions using them.

This means that if you already marked your controller methods with JAX-RS annotations and used a framework like Jersey or RestEasy, you can easily switch to serverless architecture with very little modifications in your code. You would have to change only your build mechanism and replace your preferred JAX-RS implementation with Lambada Framework.

In the following example, you see how to deploy a very basic REST API to Lambda.

1. First, clone the example project to your local directory:
   git clone https://github.com/lambadaframework/lambadaframework-boilerplate

1. This project has a pom.xml file with some configuration options. You must change the deployment.bucket option; other changes are up to you. The Lambada Framework creates this bucket in your account if it does not exists and it uses that bucket during your project's lifetime. S3 bucket names are global and must be unique, so you must pick a name which is not taken by any one else.

2. Make sure that the default AWS profile installed in your system has administrator privileges, or at least the following IAM policy:

     {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Action": [
                        "cloudformation:*",
                        "s3:*",
                        "lambda:*",
                        "execute-api:*",
                        "apigateway:*",
                        "iam:*",
                        "ec2:DescribeSecurityGroups",
                        "ec2:DescribeVpcs",
                        "ec2:DescribeSubnets"
                    ],
                    "Resource": [
                        "*"
                    ]
                }
            ]
        }

1. Now you are all set. In the root directory of your project, fire the following command:

   mvn deploy

Your project compiles to a fat JAR with all its dependencies and is deployed to Lambda. After the JAR file is on the S3 bucket, Lambada scans that for supported JAX-RS annotations in your code and creates the necessary API Gateway endpoints. At the end of the process, the URL of your API is printed on the screen. You can navigate to this URL and explore your API using the AWS Management Console to see which resources and methods are created.

Lambada Framework is under development and support for missing JAX-RS annotations are being added. Follow the Lambada Framework GitHub page for the newest features and feel free to submit any issue or contributions.

Happy serverless computing!

Carlos Sanchiz Sr. Solutions Architect

Mike Deck Partner Solutions Architect

Reputation in the email world is critical to achieve reasonable deliverability rates (the percentage of emails that arrive to inboxes); if you fall under certain levels, your emails end up in the spam folder or rejected by the email servers. To keep these numbers high, you have to constantly improve your email quality, but most importantly, you have to take action when a delivery fails or a recipient doesn't want to receive your email.

Back in 2012, we showed you how to automate the process of handling bounces and complaints with an scheduled task, using Amazon SNS, Amazon SQS, Amazon EC2, and some C# code. We have released many AWS services since then, so this post shows a different approach towards the same goal of a clean and healthy email database.

To set a little bit of context about bounces and complaints processing, I'm reusing some of the previous post:

Amazon SES assigns a unique message ID to each email that you successfully submit to send. When Amazon SES receives a bounce or complaint message from an ISP, we forward the feedback message to you. The format of bounce and complaint messages varies between ISPs, but Amazon SES interprets these messages and, if you choose to set up Amazon SNS topics for them, categorizes them into JSON objects.

Amazon SES will categorize your hard bounces into two types: permanent and transient. A permanent bounce indicates that you should never send to that recipient again. A transient bounce indicates that the recipient's ISP is not accepting messages for that particular recipient at that time and you can retry delivery in the future. The amount of time you should wait before resending to the address that generated the transient bounce depends on the transient bounce type. Certain transient bounces require manual intervention before the message can be delivered (e.g., message too large or content error). If the bounce type is undetermined, you should manually review the bounce and act accordingly.

A complaint indicates the recipient does not want the email that you sent them. When we receive a complaint, we want to remove the recipient addresses from our list.

In this post, we show you how to use AWS Lambda functions to receive SES notifications from the feedback loop from ISPs email servers via Amazon SNS and update an Amazon DynamoDB table with your email database.

Here is a high-level overview of the architecture:

Using the combination of Lambda, SNS and DynamoDB frees you from the operational overhead of having to run servers and maintain them. You focus on your application logic and AWS handles the undifferentiating heavy lifting behind the operations, scalability, and high availability.

Workflow

1. Create the SNS topic to receive the SES bounces, deliveries and complaints.
2. Create the DynamoDB table to use for our email database.
3. Create the Lambda function to process the bounces, deliveries and complaints and subscribe it to the SNS topic
4. Test & start emailing!

Create an SNS topic

First, create an SNS topic named "ses-notifications". You subscribe your Lambda function to the topic later.

Create a DynamoDB table

Create a simple DynamoDB table called "mailing" to store the email database. Use the UserId (email address) as the partition key.

Create the Lambda function

Set up your Lambda function that will process all the notifications coming from SES through your SNS topic.

Note: This post uses Node.js 4.3 as the Lambda runtime but at the time of publication, you can also use Python 2.7, Java 8 or Node.js 0.10.

For the Lambda function code, I used the recently published blueprint (ses-notification-nodejs) and adapted it to work with the DynamoDB table. The following code has the modifications highlighted:

'use strict';
console.log('Loading function');

let doc = require('dynamodb-doc');
let dynamo = new doc.DynamoDB();
let tableName = 'mailing';

exports.handler = (event, context, callback) => {
    //console.log('Received event:', JSON.stringify(event, null, 2));
    const message = JSON.parse(event.Records[0].Sns.Message);

    switch(message.notificationType) {
        case "Bounce":
            handleBounce(message);
            break;
        case "Complaint":
            handleComplaint(message);
            break;
        case "Delivery":
            handleDelivery(message);
            break;
        default:
            callback("Unknown notification type: " + message.notificationType);
    }
};

function handleBounce(message) {
    const messageId = message.mail.messageId;
    const addresses = message.bounce.bouncedRecipients.map(function(recipient){
        return recipient.emailAddress;
    });
    const bounceType = message.bounce.bounceType;

    console.log("Message " + messageId + " bounced when sending to " + addresses.join(", ") + ". Bounce type: " + bounceType);

    for (var i=0; i<addresses.length; i++){
        writeDDB(addresses[i], message, tableName, "disable");
    }
}

function handleComplaint(message) {
    const messageId = message.mail.messageId;
    const addresses = message.complaint.complainedRecipients.map(function(recipient){
        return recipient.emailAddress;
    });

    console.log("A complaint was reported by " + addresses.join(", ") + " for message " + messageId + ".");

    for (var i=0; i<addresses.length; i++){
        writeDDB(addresses[i], message, tableName, "disable");
    }
}

function handleDelivery(message) {
    const messageId = message.mail.messageId;
    const deliveryTimestamp = message.delivery.timestamp;
    const addresses = message.delivery.recipients;

    console.log("Message " + messageId + " was delivered successfully at " + deliveryTimestamp + ".");

    for (var i=0; i<addresses.length; i++){
        writeDDB(addresses[i], message, tableName, "enable");
    }
}

function writeDDB(id, payload, tableName, status) {
    const item = {
            UserId: id,
            notificationType: payload.notificationType,
            from: payload.mail.source,
            timestamp: payload.mail.timestamp,
            state: status
        };
    const params = {
            TableName:tableName,
            Item: item
        };
    dynamo.putItem(params,function(err,data){
            if (err) console.log(err);
            else console.log(data);
    });
}

Assign the function a role with execute and DynamoDB permissions so it can run and update the DynamoDB table accordingly and use index.handler as the function Handler.

This is the lambda_dynamo IAM role policy to use for the three functions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1428341300017",
            "Action": [
                "dynamodb:PutItem",
                "dynamodb:UpdateItem"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:dynamodb:us-east-1:ACCOUNT-ID:table/mailing"
        },
        {
            "Sid": "",
            "Resource": "*",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Effect": "Allow"
        }
    ]
}

You also set the corresponding SNS topic as an event source so that the Lambda function is triggered when notifications arrive.

Test the Lambda function

After everything is in place, it's time to test. Publish notifications to your SNS topics using Amazon SNS notification examples for Amazon SES, and see how your DynamoDB table is updated by your Lambda functions.

Here's an example of publishing a complaint notification to the ses-complaints-topic SNS topic using the CLI:

$ aws sns publish --topic-arn "arn:aws:sns:us-east-1:xxxxxxxxxxx:ses-notifications" --message file://message_complaints.txt --subject Test --region us-east-1

{
    "MessageId": "f7f5ad2d-a268-548d-a45c-e28e7624a64d"
}

$ cat message_complaints.txt

{
      "notificationType":"Complaint",
      "complaint":{
         "userAgent":"Comcast Feedback Loop (V0.01)",
         "complainedRecipients":[
            {
               "emailAddress":"recipient1@example.com"
            }
         ],
         "complaintFeedbackType":"abuse",
         "arrivalDate":"2009-12-03T04:24:21.000-05:00",
         "timestamp":"2012-05-25T14:59:38.623-07:00",
         "feedbackId":"000001378603177f-18c07c78-fa81-4a58-9dd1-fedc3cb8f49a-000000"
      },
      "mail":{
         "timestamp":"2012-05-25T14:59:38.623-07:00",
     "messageId":"000001378603177f-7a5433e7-8edb-42ae-af10-f0181f34d6ee-000000",
         "source":"email_1337983178623@amazon.com",
         "sourceArn": "arn:aws:sns:us-east-1:XXXXXXXXXXXX:ses-notifications",
         "sendingAccountId":"XXXXXXXXXXXX",
         "destination":[
            "recipient1@example.com",
            "recipient2@example.com",
            "recipient3@example.com",
            "recipient4@example.com"
         ]
      }
   }

And here is what you'd start seeing coming in your DynamoDB items list:

After you are done with your tests, you can point your SES notifications to the SNS topic you created and start sending emails.

Conclusion

In this post, we showed how you can use AWS Lambda, Amazon SNS, and Amazon DynamoDB to keep a healthy email database, have a good email sending score, and of course, do all of it without servers to maintain or scale.

While you're at it, why not expose your DynamoDB table with your email database using Amazon API Gateway? For more information, see Using Amazon API Gateway as a proxy for DynamoDB.

If you have questions or suggestions, please comment below.

This is a guest post from my colleagues Ryosuke Iwanaga and Prahlad Rao.

————————

Developers building and managing microservices and containerized applications using Docker containers require a secure, scalable repository to store and manage Docker images. In order to securely access the repository, proper authentication from the Docker client to the repository is important, but re-authenticating or refreshing authentication token every few hours often can be cumbersome.

This post walks you through a quick overview of Amazon ECR and how deploying Amazon ECR Docker Credential Helper can automate authentication token refresh on Docker push/pull requests.

Overview of Amazon ECS and Amazon ECR
Amazon ECS is a highly scalable, fast container management service that makes it easy to run and manage Docker containers on a cluster of Amazon EC2 instances and eliminates the need to operate your own cluster management or worry about scaling management infrastructure.

In order to reliably store Docker images on AWS, ECR provides a managed Docker registry service that is secure, scalable, and reliable. ECR is a private Docker repository with resource-based permissions using IAM so that users or EC2 instances can access repositories and images through the Docker CLI to push, pull, and manage images.

Manual ECR authentication with the Docker CLI
Most commonly, developers use Docker CLI to push and pull images or automate as part of a CI/CD workflow. Because Docker CLI does not support standard AWS authentication methods, client authentication must be handled so that ECR knows who is requesting to push or pull an image.

This can be done with a docker login command to authenticate to an ECR registry that provides an authorization token valid for 12 hours. One of the reasons for the 12-hour validity and subsequent necessary token refresh is that the Docker credentials are stored in a plain-text file and can be accessed if the system is compromised, which essentially gives access to the images. Authenticating every 12 hours ensures appropriate token rotation to protect against misuse.

If you’re using the AWS CLI, you can use a simpler get-login command which retrieves the token, decodes it, and converts into a docker login command for you. An example for the default registry associated with the account is shown below:

$ aws ecr get-login
docker login –u AWS –p password –e none https://aws_account_id.dkr.ecr.us-east-1.amazonaws.com

To access other account registries, use the -registry-ids <aws_account_id> option.

As you can see, the resulting output is a docker login command that you can use to authenticate your Docker client to your ECR registry. The generated token is valid for 12 hours, which means developers running and managing container images have to re-authenticate every 12 hours manually, or script it to generate a new token, which can be somewhat cumbersome in a CI/CD environment. For example if you’re using Jenkins to build and push docker images to ECR, you have to set up Jenkins instances to re-authenticate using get-login to ECR every 12 hours.

If you want a programmatic approach, you can use GetAuthorizationToken from the AWS SDK to fetch credentials for Docker. GetAuthorizationToken returns an authorization token of a base64-encoded string that can be decoded into username and password with “AWS” as username and temporary token as password.

It’s important to note that when executing docker login commands, the command string can be visible by other users on the system in a process list, e.g., ps –e, meaning other users can view authentication credentials to gain push and pull access to repositories. To avoid this, you can interactively log in by omitting the –p password option and enter password only when prompted. Overall, this may add additional overhead in a continuous development environment where developers need to worry about re-authentication every few hours.

Amazon ECR Docker Credential Helper
This is where Amazon ECR Docker Credential Helper makes it easy for developers to use ECR without the need to use docker login or write logic to refresh tokens and provide transparent access to ECR repositories.

Credential Helper helps developers in a continuous development environment to automate the authentication process to ECR repositories without having to regenerate tokens every 12 hours. In addition, Credential Helper also provides token caching under the hood so you don’t have to worry about getting throttled or writing additional logic.

You can access Credential Helper in the amazon-ecr-credential-helper GitHub repository.

Using Credential Helper on Linux/Mac and Windows
The prerequisites include:

• Docker 1.11 or above installed on your system
• AWS credentials available in one of the standard locations:
• ~/.aws/credentials file
• AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables
• IAM role for Amazon EC2

First, build a binary for your client machine. Although you can do it with your own Go environment, we also provide a way to build it inside a Docker container without installing Go by yourself. To build by container, just type make docker on the root directory of the repository. It will run a container FROM go image and build the binary on the mounted volume. After that, you can see it at /bin/local/docker-credential-ecr-login.

Note: You need to run this with the local Docker engine as the remote Docker Engine can’t mount your local volume.

You can also build the binary cross compiled:

• To build a Mac binary, use make docker TARGET_GOOS=darwin
• To build a Windows binary, use make docker TARGET_GOOS=windows

With these commands, Go builds the binary for the target OS inside the Linux container.

The last thing you need to do is create a Docker configuration file for the helper. Put the file under ~/.docker/config.json or C:\Users\bob\.docker\config.json with the following content:

{
    "credsStore": "ecr-login"
}

Now, you can use the docker command to interact with ECR without docker login. When you type docker push/pull YOUR_ECR_IMAGE_ID, Credential Helper is called and communicates with the ECR endpoint to get the Docker credentials. Because it automatically detects the proper region from the image ID, you don’t have to worry about it.

Using Credential Helper with Jenkins
One of the common customer deployment patterns with ECS and ECR is integrating with existing CI/CD tools like Jenkins. Using Credential Helper, your Docker CI/CD setup with Jenkins is much simpler and more reliable.
To set up ECR as a Docker image repository for Jenkins and configure Credential Helper:

• Ensure that your Jenkins instance has the proper AWS credentials to pull/push with your ECR repository. These can be in the form of environment variables, a shared credential file, or an instance profile.
• Place docker-credential-ecr-login binary at one of directories in $PATH.
• Write the Docker configuration file under the home directory of the Jenkins user, for example, /var/lib/jenkins/.docker/config.json.
• Install the Docker Build and Publish plugin and make sure that the jenkins user can contact the Docker daemon.

Then, create a project with a build step, as in the following screenshot:

Now Jenkins can push/pull images to the ECR registry without needing to refresh tokens, just like your previous Docker CLI experience.

Conclusion
The Amazon ECR Docker Credential Helper provides a very efficient way to access ECR repositories. It is transparent so that you no longer need to recall this helper after setup. This tool is hosted on GitHub and we welcome your feedback and pull requests.

If you have any questions or suggestions, please comment below.

Containers make it easy to package and share applications but they often run on a shared cluster. So how do you access your application logs for debugging? Fortunately, Docker provides a log driver that lets you send container logs to a central log service, such as Splunk or Amazon CloudWatch Logs.

Centralized logging has multiple benefits: your Amazon EC2 instance’s disk space isn’t being consumed by logs and log services often include additional capabilities that are useful for operations. For example, CloudWatch Logs includes the ability to create metrics filters that can alarm when there are too many errors and integrates with Amazon Elasticsearch Service and Kibana to enable you to perform powerful queries and analysis. This post shows how to configure Amazon ECS and CloudWatch Logs.

Step 1: Create a CloudWatch Log group

Navigate to the CloudWatch console and choose Logs. On the Actions menu, choose Create log group.

Step 2: Create an ECS task definition

The following steps assume you already have an ECS cluster created. If you do not, go through the ECS first run wizard.

A task definition defines the containers you are running and the log driver options. Navigate to the ECS console, choose Task Definitions and Create new Task Definition. Set the task definition Name and choose Add container. Set the container name, image, memory, and cpu values. In the Storage and Logging section, choose the awslogs log driver. Set the awslogs-group with the name you set in step 1. Set the awslogs-region to the region in which your task will run. Set the awslogs-stream-prefix to a custom prefix that will identify the set of logs you are streaming, such as your application’s name.

The awslogs-stream-prefix was recently added to give you the ability to associate a log stream with the ECS task ID and container name. Previously, the log stream was named with the Docker container ID, which made it hard to associate with the task. If there was an error in a log, there was no direct way to find what container was having the problem. Now, the CloudWatch log stream name includes your custom prefix, the container name, and the task ID to make it simple to associate logs with a task’s containers.
Here is a sample task definition JSON of an NGINX server that displays a welcome message:

{
    "networkMode": "bridge",
    "taskRoleArn": null,
    "containerDefinitions": [
        {
            "memory": 300,
            "portMappings": [
                {
                    "hostPort": 80,
                    "containerPort": 80,
                    "protocol": "tcp"
                }
            ],
            "essential": true,
            "entryPoint": [
                "sh",
                "-c"
            ],
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-group": "awslogs-test",
                    "awslogs-region": "us-west-2",
                    "awslogs-stream-prefix": "nginx"
                }
            },
            "name": "simple-app",
            "image": "httpd:2.4",
            "command": [
                "/bin/sh -c \"echo 'Congratulations! Your application is now running on a container in Amazon ECS.'  > /usr/local/apache2/htdocs/index.html && httpd-foreground\""
            ],
            "cpu": 10
        }
    ],
    "family": "cw-logs-example"
}

Step 3: Run the task

In the ECS console, choose Clusters. Select your cluster, then choose the Tasks tab. Choose Run new task and in the Task definition list, select the task definition that you created in step 2. Choose Run Task.

You will see your task in the PENDING state. Select the task to open the detail view. Refresh your task’s detail view until the task gets to the RUNNING state.

Step 4: Generate logs

If you’re using the sample task definition, NGINX will have already sent an initialization message to the log stream. You can also connect with the web server to generate additional log messages.

Step 5: View the log

The task view now includes a link to the log stream. Select the link and navigate to the CloudWatch console. The log stream name includes the prefix that you specified in the task definition, the container name, and the ECS task ID (nginx/simple-app/600e016a-9301-4f81-90b2-6bfd0ad2d975). This makes it easy to find the log stream from the ECS task and find the task from the log stream.

Cleanup

When you are done, you can stop the task in the ECS console and remove the log stream in the CloudWatch console.

Conclusion

We hope you find these improvements useful. You can also use CloudWatch Logs for ECS agent and Docker logs. For more information, see the ECS documentation. If you have suggestions or questions, please comment below.

Julien Lepine
Solutions Architect

As developers, one of our top priorities is to build reliable systems; this is a core pillar of the AWS Well Architected Framework. A common pattern to fulfill this goal is to have an architecture built around loosely coupled components.

Amazon Kinesis Streams offers an excellent answer for this, as the events generated can be consumed independently by multiple consumers and remain available for 1 to 7 days. Building an Amazon Kinesis consumer application is done by leveraging the Amazon Kinesis Client Library (KCL) or native integration with AWS Lambda.

As I was speaking with other developers and customers about their use of Amazon Kinesis, there are a few patterns that came up. This post addresses those common patterns.

Protecting streams

Amazon Kinesis has made the implementation of event buses easy and inexpensive, so that applications can send meaningful information to their surrounding ecosystem. As your applications grow and get more usage within your company, more teams will want to consume the data generated, even probably external parties such as business partners or customers.

When the applications get more usage, some concerns may arise:

• When a new consumer starts (or re-starts after some maintenance), it needs to read a lot of data from the stream (its backlog) in a short amount of time in order to get up to speed
• A customer may start many consumers at the same time, reading a lot of events in parallel or having a high call rate to Amazon Kinesis
• A consumer may have an issue (such as infinite loop, retry error) that causes it to call Amazon Kinesis at an extremely high rate

These cases may lead to a depletion of the resources available in your stream, and that could potentially impact all your consumers.

Managing the increased load can be done by leveraging the scale-out model of Amazon Kinesis through the addition of shards to an existing stream. Each shard adds both input (ingestion) and output (consumption) capacity to your stream:

• 1000 write records and up to 1 megabyte per second for ingesting events
• 5 read transactions and up to 2 megabytes per second for consuming events

Avoiding these scenarios could be done by scaling-out your streams, and provisioning for peak, but that would create inefficiencies and may not even fully protect your consumers from the behavior of others.

What becomes apparent in these cases is the impact that a single failing consumer may have on all other consumers, a symptom described as the “noisy neighbor”, or managing the blast radius of your system. The key point is to limit the impact that a single consumer can have on others.

A solution is to compartmentalize your platform: this method consists of creating multiple streams and then creating groups of consumers that share the same stream. This gives you the possibility to limit the impact a single consumer can have on its neighbors, and potentially to propose a model where some customers have a dedicated stream.

You can build an Amazon Kinesis consumer application (via the KCL or Lambda) that reads a source stream and sends the messages to the “contained” streams that the actual consumers will use.

Transforming streams

Another use case I see from customers is the need to transfer the data in their stream to other services:

• Some applications may have limitations in their ability to receive or process the events
• They may not have connectors to Amazon Kinesis, and only support Amazon SQS
• They may only support a push model, where their APIs need to be called directly when a message arrives
• Some analytics/caching/search may be needed on the events generated
• Data may need to be archived or sent to a data warehouse engine

There are many other cases, but the core need is having the ability to get the data from Amazon Kinesis into other platforms.

The solution for these use cases is to build an Amazon Kinesis consumer application that reads a stream and prepares these messages for other services.

Sharing data with external parties

The final request I have seen is the possibility to process a stream from a different AWS account or region. While you can give access to your resources to an external AWS account through cross-account IAM roles, that feature requires development and is not supported natively by some services. For example, you cannot subscribe a Lambda function to a stream in a different AWS account or region.

The solution is to replicate the Amazon Kinesis stream or the events to another environment (AWS account, region, or service).

This can be done one time through an Amazon Kinesis consumer application that reads a stream and forwards the events to the remote environment.

Solution: A Lambda-based fan-out function

These three major needs have a common solution: the deployment of an Amazon Kinesis consumer application that listens to a stream and is able to send messages to other instances of Amazon Kinesis, services, or environments (AWS accounts or regions).

In the aws-lambda-fanout GitHub repository, you’ll find a Lambda function that specifically supports this scenario. This function is made to forward incoming messages from Amazon Kinesis or DynamoDB Streams.

The architecture of the function is made to be simple and extensible, with one core file fanout.js that loads modules for the different providers. The currently supported providers are as follows:

• Amazon SNS
• Amazon SQS
• Amazon Elasticsearch Service
• Amazon Kinesis Streams
• Amazon Kinesis Firehose
• AWS IoT
• AWS Lambda
• Amazon ElastiCache for Memcached
• Amazon ElastiCache for Redis

The function is built to support multiple inputs:

• Amazon Kinesis streams
• Amazon Kinesis streams containing Amazon Kinesis Producer Library (KPL) records
• DynamoDB Streams records

It relies on Lambda for a fully-managed environment where scaling, logging, and monitoring are automated by the platform. It also supports Lambda functions in a VPC for Amazon ElastiCache.

The configuration is stored in a DynamoDB table, and associates the output configuration with each function. This table has a simple schema:

• sourceArn (Partition Key): The Amazon Resource Name (ARN) of the input Amazon Kinesis stream
• id [String]: The name of the mapping
• type [String]: The destination type
• destination [String]: The ARN or name of the destination
• active [Boolean]: Whether that mapping is active

Depending on the target, some other properties are also stored.

The function can also group records together for services that don’t initially support it, such as Amazon SQS, Amazon SNS, or AWS IoT. Amazon DynamoDB Streams records can also be transformed to plain JSON objects to simplify management in later stages. The function comes with a Bash-based command line Interface to make the deployment and management easier.

As an example, the following lines deploy the function, which registers a mapping from one stream (inputStream) to another (outputStream).

./fanout deploy --function fanout

./fanout register kinesis --function fanout --source-type kinesis --source inputStream --id target1 --destination outputStream --active true

./fanout hook --function fanout --source-type kinesis --source inputStream

Summary

There are many options available for you to forward your events from one service or environment to another. For more information about this topic, see Using AWS Lambda with Amazon Kinesis. Happy eventing!

If you have questions or suggestions, please comment below.

Ed Lima
Cloud Support Engineer

Does your IoT device fleet scale to hundreds or thousands of devices? Do you find it somewhat challenging to retrieve the details for multiple devices? AWS IoT provides a platform to connect those devices and build a scalable solution for your Internet of Things workloads.

Out of the box, the AWS IoT console gives you your own searchable device registry with access to the device state and information about device shadows. You can enhance and customize the service using AWS Lambda and Amazon DynamoDB to build a serverless backend with a customizable device database that can be used to store useful information about the devices as well as helping to track what devices are activated with an activation code, if required.

You can use DynamoDB to extend the AWS IoT internal device registry to help manage the device fleet, as well as storing specific additional data about each device. Lambda provides the link between AWS IoT and DynamoDB allowing you to add, update, and query your new device database backend.

In this post, you learn how to use AWS IoT rules to trigger specific device registration logic using Lamba in order to populate a DynamoDB table. You then use a second Lambda function to search the database for a specific device serial number and a randomly generated activation code to activate the device and register the email of the device owner in the same table. After you’re done, you’ll have a fully functional serverless IoT backend, allowing you to focus on your own IoT solution and logic instead of managing the infrastructure to do so.

Prerequisites

You must have the following before you can create and deploy this framework:

• An AWS account
• An IAM user with permissions to create AWS resources (AWS IoT things and rules, Lambda functions, DynamoDB tables, IAM policies and roles, etc.)
• JS and the AWS SDK for JavaScript installed locally to test the deployment

Building a backend

In this post, I assume that you have some basic knowledge about the services involved. If not, you can review the documentation:

• Creating Lambda functions
• Creating DynamoDB tables
• Overview of AWS IoT
• Configuring AWS IoT rules to trigger newly-created Lambda functions

For this use case, imagine that you have a fleet of devices called “myThing”. These devices can be anything: a smart lightbulb, smart hub, Internet-connected robot, music player, smart thermostat, or anything with specific sensors that can be managed using AWS IoT.

When you create a myThing device, there is some specific information that you want to be available in your database, namely:

• Client ID
• Serial number
• Activation code
• Activation status
• Device name
• Device type
• Owner email
• AWS IoT endpoint

The following is a sample payload with details of a single myThing device to be sent to a specific MQTT topic, which triggers an IoT rule. The data is in a format that AWS IoT can understand, good old JSON. For example:

{
  "clientId": "ID-91B2F06B3F05",
  "serialNumber": "SN-D7F3C8947867",
  "activationCode": "AC-9BE75CD0F1543D44C9AB",
  "activated": "false",
  "device": "myThing1",
  "type": "MySmartIoTDevice",
  "email": "not@registered.yet",
  "endpoint": "<endpoint prefix>.iot.<region>.amazonaws.com"
}

The rule then invokes the first Lambda function, which you create now. Open the Lambda console, choose Create a Lambda function , and follow the steps. Here’s the code:

console.log('Loading function');
var AWS = require('aws-sdk');
var dynamo = new AWS.DynamoDB.DocumentClient();
var table = "iotCatalog";

exports.handler = function(event, context) {
    //console.log('Received event:', JSON.stringify(event, null, 2));
   var params = {
    TableName:table,
    Item:{
        "serialNumber": event.serialNumber,
        "clientId": event.clientId,
        "device": event.device,
        "endpoint": event.endpoint,
        "type": event.type,
        "certificateId": event.certificateId,
        "activationCode": event.activationCode,
        "activated": event.activated,
        "email": event.email
        }
    };

    console.log("Adding a new IoT device...");
    dynamo.put(params, function(err, data) {
        if (err) {
            console.error("Unable to add device. Error JSON:", JSON.stringify(err, null, 2));
            context.fail();
        } else {
            console.log("Added device:", JSON.stringify(data, null, 2));
            context.succeed();
        }
    });
}

The function adds an item to a DynamoDB database called iotCatalog based on events like the JSON data provided earlier. You now need to create the database as well as making sure the Lambda function has permissions to add items to the DynamoDB table, by configuring it with the appropriate execution role.

Open the DynamoDB console, choose Create table and follow the steps. For this table, use the following details.

The serial number uniquely identifies your device; if, for instance, it is a smart hub that has different client devices connecting to it, use the client ID as the sort key.

The backend is good to go! You just need to make the new resources work together; for that, you configure an IoT rule to do so.

On the AWS IoT console, choose Create a resource and Create a rule , and use the following settings to point the rule to your newly-created Lambda function, also called iotCatalog.

After creating the rule, AWS IoT adds permissions on the background to allow it to trigger the Lambda function whenever a message is published to the MQTT topic called registration. You can use the following Node.js deployment code to test:

var AWS = require('aws-sdk');
AWS.config.region = 'ap-northeast-1';

var crypto = require('crypto');
var endpoint = "<endpoint prefix>.iot.<region>.amazonaws.com";
var iot = new AWS.Iot();
var iotdata = new AWS.IotData({endpoint: endpoint});
var topic = "registration";
var type = "MySmartIoTDevice"

//Create 50 AWS IoT Things
for(var i = 1; i < 51; i++) {
  var serialNumber = "SN-"+crypto.randomBytes(Math.ceil(12/2)).toString('hex').slice(0,15).toUpperCase();
  var clientId = "ID-"+crypto.randomBytes(Math.ceil(12/2)).toString('hex').slice(0,12).toUpperCase();
  var activationCode = "AC-"+crypto.randomBytes(Math.ceil(20/2)).toString('hex').slice(0,20).toUpperCase();
  var thing = "myThing"+i.toString();
  var thingParams = {
    thingName: thing
  };
  
  iot.createThing(thingParams).on('success', function(response) {
    //Thing Created!
  }).on('error', function(response) {
    console.log(response);
  }).send();

  //Publish JSON to Registration Topic

  var registrationData = '{\n \"serialNumber\": \"'+serialNumber+'\",\n \"clientId\": \"'+clientId+'\",\n \"device\": \"'+thing+'\",\n \"endpoint\": \"'+endpoint+'\",\n\"type\": \"'+type+'\",\n \"activationCode\": \"'+activationCode+'\",\n \"activated\": \"false\",\n \"email\": \"not@registered.yet\" \n}';

  var registrationParams = {
    topic: topic,
    payload: registrationData,
    qos: 0
  };

  iotdata.publish(registrationParams, function(err, data) {
    if (err) console.log(err, err.stack); // an error occurred
    // else Published Successfully!
  });
  setTimeout(function(){},50);
}

//Checking all devices were created

iot.listThings().on('success', function(response) {
  var things = response.data.things;
  var myThings = [];
  for(var i = 0; i < things.length; i++) {
    if (things[i].thingName.includes("myThing")){
      myThings[i]=things[i].thingName;
    }
  }

  if (myThings.length = 50){
    console.log("myThing1 to 50 created and registered!");
  }
}).on('error', function(response) {
  console.log(response);
}).send();

console.log("Registration data on the way to Lambda and DynamoDB");

The code above creates 50 IoT things in AWS IoT and generate random client IDs, serial numbers, and activation codes for each device. It then publishes the device data as a JSON payload to the IoT topic accordingly, which in turn triggers the Lambda function:

And here it is! The function was triggered successfully by your IoT rule and created your database of IoT devices with all the custom information you need. You can query the database to find your things and any other details related to them.

In the AWS IoT console, the newly-created things are also available in the thing registry.

Now you can create certificates, policies, attach them to each “myThing” AWS IoT Thing then install each certificate as you provision the physical devices.

Activation and registration logic

However, you’re not done yet…. What if you want to activate a device in the field with the pre-generated activation code as well as register the email details of whoever activated the device?

You need a second Lambda function for that, with the same execution role from the first function (Basic with DynamoDB). Here’s the code:

console.log('Loading function');

var AWS = require('aws-sdk');
var dynamo = new AWS.DynamoDB.DocumentClient();
var table = "iotCatalog";

exports.handler = function(event, context) {
    //console.log('Received event:', JSON.stringify(event, null, 2));

   var params = {
    TableName:table,
    Key:{
        "serialNumber": event.serialNumber,
        "clientId": event.clientId,
        }
    };

    console.log("Gettings IoT device details...");
    dynamo.get(params, function(err, data) {
    if (err) {
        console.error("Unable to get device details. Error JSON:", JSON.stringify(err, null, 2));
        context.fail();
    } else {
        console.log("Device data:", JSON.stringify(data, null, 2));
        console.log(data.Item.activationCode);
        if (data.Item.activationCode == event.activationCode){
            console.log("Valid Activation Code! Proceed to register owner e-mail and update activation status");
            var params = {
                TableName:table,
                Key:{
                    "serialNumber": event.serialNumber,
                    "clientId": event.clientId,
                },
                UpdateExpression: "set email = :val1, activated = :val2",
                ExpressionAttributeValues:{
                    ":val1": event.email,
                    ":val2": "true"
                },
                ReturnValues:"UPDATED\_NEW"
            };
            dynamo.update(params, function(err, data) {
                if (err) {
                    console.error("Unable to update item. Error JSON:", JSON.stringify(err, null, 2));
                    context.fail();
                } else {
                    console.log("Device now active!", JSON.stringify(data, null, 2));
                    context.succeed("Device now active! Your e-mail is now registered as device owner, thank you for activating your Smart IoT Device!");
                }
            });
        } else {
            context.fail("Activation Code Invalid");
        }
    }
});
}

The function needs just a small subset of the data used earlier:

{
  "clientId": "ID-91B2F06B3F05",
  "serialNumber": "SN-D7F3C8947867",
  "activationCode": "AC-9BE75CD0F1543D44C9AB",
  "email": "verified@registered.iot"
}

Lambda uses the hash and range keys (serialNumber and clientId) to query the database and compare the database current pre-generated activation code to a code that is supplied by the device owner along with their email address. If the activation code matches the one from the database, the activation status and email details are updated in DynamoDB accordingly. If not, the user gets an error message stating that the code is invalid.

You can turn it into an API with Amazon API Gateway. In order to do so, go to the Lambda function and add an API endpoint, as follows.

Now test the access to the newly-created API endpoint, using a tool such as Postman.

If an invalid code is provided, the requester gets an error message accordingly.

Back in the database, you can confirm the record was updated as required.

Cleanup

After you finish the tutorial, delete all the newly created resources (IoT things, Lambda functions, and DynamoDB table). Alternatively, you can keep the Lambda function code for future reference, as you won’t incur charges unless the functions are invoked.

Conclusion

As you can see, by leveraging the power of the AWS IoT Rules Engine, you can take advantage of the seamless integration with AWS Lambda to create a flexible and scalable IoT backend powered by Amazon DynamoDB that can be used to manage your growing Internet of Things fleet.

You can also configure an activation API to make use of the newly-created backend and activate devices as well as register email contact details from the device owner; this information could be used to get in touch with your users regarding marketing campaigns or newsletters about new products or new versions of your IoT products.

If you have questions or suggestions, please comment below.

This is a guest post from Sathiya Shunmugasundaram, Gnani Dathathreya, and Jeff Storey from the Capital One Technology team.

—————–

At Capital One, we are rapidly embracing cloud-native microservices architectures and applying them to both existing and new workloads. To advance microservices adoption, increase efficiencies of cloud resources and decouple application layer from the underlying infrastructure, we are starting to use Docker to containerize the workloads and Amazon EC2 Container Service (Amazon ECS) to manage them.

Docker enables environment consistency and allows us to spin up new containers in a location (Dev / QA / Performance / Prod) of choice in seconds vs. minutes / hours / days it used to take us in the past.

Amazon ECS gives us a platform to manage Docker containers with no hassle. We chose ECS because of its simplicity in deploying and managing containers. Our API platform is an early adopter and ECS-based deployments quickly became the norm for managing the lifecycle of stateless workloads.

Container orchestration has traditionally been challenging. With the Elastic Load Balancing Classic Load Balancer, we have had limitations routing to multiple ports on the same server and are also unable to route services based on context, which meant that we needed to use one load balancer per service. By using open source software like Consul, Nginx, and registrator it was possible to achieve dynamic service discovery and context based routing but at the cost of adding more complexity and costs for running these additional components.

This post shows how the arrival of the Application Load Balancer has significantly simplified Docker based deployments on ECS and enabled delivering microservices in the cloud with enterprise class capabilities like service discovery, health checks, and load balancing.

Overview of Application Load Balancer

With the announcement of the new Application Load Balancer, we can take advantage of several out of the box features that are readily integrated with ECS. With dynamic port mapping option for containers we can simply register a service with a load balancer, and ECS transparently manages the registration and de-registration of Docker containers. We no longer need to know the host port ahead of time, as the load balancer automatically detects it and dynamically reconfigures itself. In addition to the port mapping, we also get all the features of traditional load balancers, like health checking, connection draining and access logs to name a few.

Similar to EC2 Auto Scaling, ECS also has the ability to auto scale services based on CloudWatch alarms. This functionality is critical as it allows us to scale-in or scale-out based on demand. This feature coupled with the new Application Load Balancer gives us fully-featured container orchestration. The pace at which we can now spin up new applications on ECS has greatly improved and we can spend much less time managing orchestration tools.

The Application Load Balancer also introduces path-based routing. This feature allows us to easily map different URL paths to different services. In a traditional monolithic application, URL paths are often used to denote different parts of the application, for example, http://<apphost>.com/service1 and http://<apphost>.com/service2.

Traditionally this was done using a context root in an application server or by using different load balancers for each service. With the new path-based routing, these parts of the application can be split into individual ECS-backed services without changing the existing URL patterns. This makes migrating applications seamless, as clients can continue to call the same URLs. A large monolithic application with many subcontexts like www.example.com/orders, www.example.com/inventory can be refactored into smaller micro services and each such path can be directed to a different target group of servers such as an ECS Service with Docker containers.

Key features of Application Load Balancers include:

• Path-based routing – URL-based routing policies enable using the same ELB URL to route to different microservices
• Multiple ports routing on same server
• AWS integration – Integrated with many AWS services, such as ECS, IAM, Auto Scaling, and CloudFormation
• Application monitoring – Improved metrics and health checks for the application

Core components of Application Load Balancers include:

• Load balancer – The entry point for clients
• Listener – Listens to requests from clients on a specific protocol/port and forwards to one or more target group based on rules
• Rule – Determines how to route the request – based on path-based condition and priority matches to one or more target groups
• Target – The entity that runs the backend servers – currently EC2 is the available target group. The same EC2 instance can be registered multiple times with different ports
• Target group – Each target group identifies a set of backend servers which can be routed based on a rule. Health checks can be defined per target group. The same load balancer can have many target groups

Sample application architecture

In the following example, we show two services with three tasks deployed on two ECS container instances. This shows the ability to route to multiple ports on the same host. Also, there is only one Application Load Balancer that provides path-based routing for both ECS services, simplifying the architecture and reducing costs.

Configuring Application Load Balancers

The following steps create and configure an Application Load Balancer using the AWS Management Console. These steps can also be done using the AWS CLI.

1. Create an Application Load Balancer using the AWS Console
• Login to the EC2 Console (https://console.aws.amazon.com/ec2)
• Select Load Balancers
• Select Create Load Balancer



• Choose Application Load Balancer
2. Configure Load Balancer



• For Name, type a name for your load balancer.
• For Scheme, an Internet-facing load balancer routes requests from clients over the Internet to targets. An internal load balancer routes requests to targets using private IP addresses.
• For Listeners, the default is a listener that accepts HTTP traffic on port 80. You can keep the default listener settings, modify the protocol or port of the listener, or choose Add to add another listener.
• For VPC, select the same VPC that you used for the container instances on which you intend to run your service.
• For Available subnets, select at least two subnets from different Availability Zones, and choose the icon in the Actions column.
3. Configure Security Groups

You must assign a security group to your load balancer that allows inbound traffic to the ports that you specified for your listeners.

4. Configure Routing
• For Target group, keep the default, New target group.
• For Name, type a name for the new target group.
• Set Protocol and Port as needed.
• For Health checks, keep the default health check settings.



5. Register Targets

Your load balancer distributes traffic between the targets that are registered to its target groups. When you associate a target group to an Amazon ECS service, Amazon ECS automatically registers and deregisters containers with your target group. Because Amazon ECS handles target registration, you do not add targets to your target group at this time.



• Click Next:Review
• Click Create
6. Registering Docker containers as a target
• If you don’t already have an ECS cluster, open the Amazon ECS console first run wizard at https://console.aws.amazon.com/ecs/home#/firstRun.
• From the ECS Cluster’s Services tab, Click Create



• Provide the Task definition for service
• Provide a Service Name and number of tasks
• Click the Configure ELB button
• Choose Application Load Balancer



• Choose the ecsService Role as IAM role
• Choose the Application Load Balancer created above
• Select a container that you want the load balancer to use and click Add to ELB



• Select the target group name that was created above
• Click Save and then Create Service

Cleaning up

• Using the ECS Console, go to your Cluster and Service, update the number of tasks to 0, and then delete the service.
• Using the EC2 Console, select the Load Balancer created above and delete.

Conclusion
Docker has given our developers a simplified application automation mechanism. Amazon ECS and Application Load Balancers make it easy to deliver these applications without needing to manage dynamic service discovery, load balancing and container orchestration. Using ECS and Application Load Balancers new services can be deployed in less than 30 minutes and existing services updated with newer versions in less than a minute. ECS automatically takes care of rolling updates and Application Load Balancer takes care of registering new versions quickly and unregistering existing containers gracefully. This not only improves the agility of teams, but also reduces the overall time to market.

Ronald Widha @ronaldwidha
Partner Solutions Architect

Redirection is neither a success nor an error response. You return redirection when the requested resource resides either temporarily or permanently under a different URI. The client needs to issue subsequent calls to the new location in order to retrieve the requested resource. Even though you typically see 302 and 301 redirects when requesting text/html, these response types also apply to REST JSON endpoints.

Many of you are already familiar how to return success and error responses. Amazon API Gateway and AWS Lambda help development teams to build scalable web endpoints very quickly. In a previous post (Error Handling Patterns in Amazon API Gateway and AWS Lambda), we discussed several error handling patterns to implement an explicit contract for the types of error responses that an API can produce and how they translate into HTTP status codes. However, so far this blog has not discussed how to handle redirection.

This post shows the recommended patterns for handling redirection in your serverless API built on API Gateway and Lambda.

Routing Lambda errors to API Gateway HTTP 30x responses

In HTTP, there are several types of redirection codes. You return these status codes as part of an HTTP response to the client.

For more information about HTTP server status codes, see RFC2616 section 10.5 on the W3C website.

In this specific scenario, you would like your API Gateway method to return the following HTTP response. Note that there are two key parameters to return:

1. The status code, i.e., 301 or 302
2. The new URI of the resource

HTTP/1.1 302 Found
Content-Type: text/html
Content-Length: 503
Connection: keep-alive
Date: Fri, 08 Jul 2016 16:16:44 GMT
Location: http://www.amazon.com/

In API Gateway, AWS recommends that you model the various HTTP response types that your API method may produce, and define a mapping from the return value of your Lambda function to these HTTP responses. To do that, you need to do two things:

1. API Gateway can only map responses to different method responses on error. So even though redirection isn’t strictly a failure, you still throw an exception from Lambda to communicate back to API Gateway when it needs to issue a 30x response.
2. Unlike HTTP 200 Success or any of the HTTP Error status codes, a redirection requires two values: the status code and the location. API Gateway do not support VTL for JSON serialization in the header mappings; thus, in this case, you need to take advantage of how the Lambda runtime handles exceptions to pass these values in two separate fields back to API Gateway.

Node.JS: Using the exception name to store the redirection URI

The mapping from a Lambda function error to an API Gateway method response is defined by an integration response, which defines a selection pattern used to match the Lambda function errorMessage and routes it to an associated method response. In this example, you use the prefix-based error handling pattern:

Because you don’t have access to VTL in the API Gateway header mappings, retrieve your redirection URI from errorType.

In the Lambda function Node.JS runtime, errorType can be assigned to any value. In this case, use it to store the redirection URI. In the handler.js, you have the following:

// Returns 302 or 301
var err = new Error("HandlerDemo.ResponseFound Redirection: Resource found elsewhere");
err.name = "http://a-different-uri";
context.done(err, {});

The same technique applies to 301 permanent redirects except you replace the API Gateway regex above to detect HandlerDemo.MovedPermanently.

Node.JS: Handling other response types

In order to keep it consistent, you can leverage the same pattern for other error codes. You can leverage errorType for any end user–visible messages.

//404 error
var err = new Error(“HandlerDemo.ResponseNotFound: Resource not found.”);
err.name = “Sorry, we can't find what you're looking for. Are you sure the address is correct?";
context.done(err, {});

Just as an example, upon failure, you return a “Fail Whale” HTML webpage to the user.

HTTP 200 Success can be handled as you normally would in your handler.js:

//200 OK
context.done(null, response);

Java: Using the Java inner exception to store the redirection URI

Because Java is a strongly typed language, the errorType Lambda function return value contains the Java exception type which you cannot override (nor should you). Thus, you will be using a different property to retrieve the redirection URI:

The cause.errorMessage parameter is accessible in the Lambda function Java runtime as an inner exception.

throw new Exception(new ResponseFound("http://www.amazon.com"));

ResponseFound is a class that extends Throwable.

package HandlerDemo;

public class ResponseFound extends Throwable {
  public ResponseFound(String uri) { super(uri); }
}

The full response from Lambda received by API Gateway is the following:

{
  "errorMessage": "HandlerDemo.ResponseFound: http://www.amazon.com",
  "errorType": "java.lang.Exception",
  "stackTrace": [ ... ],
  "cause": {
    "errorMessage": "http://www.amazon.com",
    "errorType": "HandlerDemo.ResponseFound",
    "stackTrace": [ ... ]
  }
}

Because Lambda serializes the inner exception type to the outer errorMessage. you can still use the same Lambda error regex in API Gateway to detect a redirect from errorMessage.

Conclusion

In this post, I showed how to emit different HTTP responses, including a redirect response, on Amazon API Gateway and AWS Lambda for Node.JS and Java runtimes. I encourage developers to wrap these patterns into a helper class where it handles the inner working of returning a success, error, or redirection response.

One thing you may notice missing is an example of handling redirection in Python. At the time of writing, the Chalice community is considering support for this use case as part of the framework.

If you have questions or suggestions, please comment below.

Mark Statham, Senior Cloud Architect

When you implement hybrid connectivity between existing on-premises environments and AWS, there are a number of approaches to provide DNS resolution of both on-premises and VPC resources. In a hybrid scenario, you likely require resolution of on-premises resources, AWS services deployed in VPCs, AWS service endpoints, and your own resources created in your VPCs.

You can leverage Amazon Route 53 private hosted zones to provide private DNS zones for your VPC resources and dynamically register resources, as shown in a previous post, Building a Dynamic DNS for Route 53 using CloudWatch Events and Lambda.

Ultimately, this complex DNS resolution scenario requires that you deploy and manage additional DNS infrastructure, running on EC2 resources, into your VPC to handle DNS requests either from VPCs or on-premises. Whilst this is a familiar approach it adds additional cost and operational complexity, where a solution using AWS managed services can be used instead.

In this post, we explore how you can use leverage Route 53 private hosted zones with AWS Lambda and Amazon CloudWatch Events to mirror on-premises DNS zones which can then be natively resolved from within your VPCs, without the need for additional DNS forwarding resources.

Route 53 private hosted zones

Route 53 offers the convenience of domain name services without having to build a globally distributed highly reliable DNS infrastructure. It allows instances within your VPC to resolve the names of resources that run within your AWS environment. It also lets clients on the Internet resolve names of your public-facing resources. This is accomplished by querying resource record sets that reside within a Route 53 public or private hosted zone.

A private hosted zone is basically a container that holds information about how you want to route traffic for a domain and its subdomains within one or more VPCs and is only resolvable from the VPCs you specify; whereas a public hosted zone is a container that holds information about how you want to route traffic from the Internet.

Route 53 has a programmable API that can be used to automate the creation/removal of records sets which we're going leverage later in this post.

Using Lambda with VPC support and scheduled events

AWS Lambda is a compute service where you can upload your code and the service runs the code on your behalf using AWS infrastructure. You can create a Lambda function and execute it on a regular schedule. You can specify a fixed rate (for example, execute a Lambda function every hour or 15 minutes), or you can specify a cron expression. This functionality is underpinned by CloudWatch Events.

Lambda runs your function code securely within a VPC by default. However, to enable your Lambda function to access resources inside your private VPC, you must provide additional VPC-specific configuration information that includes VPC subnet IDs and security group IDs. Lambda uses this information to set up elastic network interfaces (ENIs) that enable your function to connect securely to other resources within your private VPC or reach back into your own network via AWS Direct Connect or VPN.

Each ENI is assigned a private IP address from the IP address range within the subnets that you specify, but is not assigned any public IP addresses. You cannot use an Internet gateway attached to your VPC, as that requires the ENI to have public IP addresses. Therefore, if your Lambda function requires Internet access, for example to access AWS APIs, you can use the Amazon VPC NAT gateway. Alternatively, you can leverage a proxy server to handle HTTPS calls, such as those used by the AWS SDK or CLI.

Building an example system

When you combine the power of Route 53 private hosted zones and Lambda, you can create a system that closely mimics the behavior of a stealth DNS to provide resolution of on-premises domains via VPC DNS.

For example, it is possible to schedule a Lambda function that executes every 15 minutes to perform a zone transfer from an on-premises DNS server, using a full zone transfer query (AXFR). The function can check the retrieved zone for differences from a previous version. Changes can then be populated into a Route 53 private hosted zone, which is only resolvable from within your VPCs, effectively mirroring the on-premises master to Route 53.

This then allows your resources deployed in VPC to use just VPC DNS to resolve on-premises, VPC and Internet resources records without the need for any additional forwarding infrastructure to on-premises DNS.

The following example is based on python code running as a Lambda function, invoked using CloudWatch Events with constant text to provide customizable parameters to support the mirroring of multiple zones for both forward and reverse domains.

Prerequisites for the example

Before you get started, make sure you have all the prerequisites in place including installing the AWS CLI and creating a VPC.

• Region

  Check that the region where your VPC is deployed has the Lambda and CloudWatch Events services available.

• AWS Command Line Interface (AWS CLI)

  This example makes use of the AWS CLI; however, all actions can be performed via the AWS console as well. Make sure you have the latest version installed, which provides support for creating Lambda functions in a VPC and the required IAM permissions to create resources required. For more information, see Getting Set Up with the AWS Command Line Interface.

• VPC

  For this example, create or use a VPC configured with at least two private subnets in different Availability Zones and connectivity to the source DNS server. If you are building a new VPC, see Scenario 4: VPC with a Private Subnet Only and Hardware VPN Access.

  Ensure that the VPC has the DNS resolution and DNS hostnames options set to yes, and that you have both connectivity to your source DNS server and the ability to access the AWS APIs. You can create an AWS managed NAT gateway to provide Internet access to AWS APIs or as an alternative leverage a proxy server.

  You may wish to consider creating subnets specifically for your Lambda function, allowing you to restrict the IP address ranges that need access to the source DNS server and configure network access controls accordingly.

  After the subnets are created, take note of them as you'll need them later to set up the Lambda function: they are in the format subnet-ab12cd34. You also need a security group to assign to the Lambda function; this can be the default security group for the VPC or one you create with limited outbound access to your source DNS: the format is sg-ab12cd34.

• DNS server

  You need to make sure that you modify DNS zone transfer settings so that your DNS server accepts AXFR queries from the Lambda function. Also, ensure that security groups or firewall policies allow connection via TCP port 53 from the VPC subnet IP ranges created above.

Setting up the example Lambda function

Before you get started, it's important to understand how the Lambda function works and interacts with the other AWS services and your network resources:

1. The Lambda function is invoked by CloudWatch Events and configured based on a JSON string passed to the function. This sets a number of parameters, including the DNS domain, source DNS server, and Route 53 zone ID. This allows a single Lambda function to be reused for multiple zones.
2. A new ENI is created in your VPC subnets and attached to the Lambda function; this allows your function to access your internal network resources based on the security group that you defined.
3. The Lambda function then transfers the source DNS zone from the IP specified in the JSON parameters. You need to ensure that your DNS server is configured to allow full zone transfers and allow AXFR queries to your DNS server, which happens over TCP port 53.
4. The Route 53 DNS zone is retrieved via API.
5. The two zone files are compared; the resulting differences are returned as a set of actions to be performed against Route 53.
6. Updates to the Route 53 zone are made via API and, finally, the SOA is updated to match the source version.

You're now ready to set up the example using the following instructions.

Step 1 – Create a Route 53 hosted zone

Before you create the Lambda function, there needs to be a target Route 53 hosted zone to mirror the DNS zone records into. This can either be a public or private zone; however, for the purposes of this example, you will create a private hosted zone that only responds to queries from the VPC you specify.

To create a Route 53 private hosted zone associated with your VPC, provide the region and VPC ID as part of the following command:

aws route53 create-hosted-zone \
--name <domainname> \
--vpc VPCRegion=<region>,VPCId=<vpc-aa11bb22> \
--caller-reference mirror-dns-lambda \
--hosted-zone-config Comment="My DNS Domain"

Save the HostedZone Id returned, since you will need it for future steps.

Step 2 – Create an IAM role for the Lambda function

In this step, you use the AWS CLI to create the Identity and Access Management (IAM) role that the Lambda function assumes when the function is invoked. You need to create an IAM policy with the required permissions and then attach this policy to the role.

Download the mirror-dns-policy.json and mirror-dns-trust.json files from the aws-lambda-ddns-function AWS Labs GitHub repo.

mirror-dns-policy.json

The policy includes EC2 permissions to create and manage ENIs required for the Lambda function to access your VPC, and Route 53 permissions to list and create resource records. The policy also allows the function to create log groups and log events as per standard Lambda functions.

{
    "Version": "2012-10-17",
    "Statement": [{
        "Effect": "Allow",
        "Action": [
            "logs:CreateLogGroup",
            "logs:CreateLogStream",
            "logs:PutLogEvents"
        ],
        "Resource": "arn:aws:logs:*:*:*"
    }, {

        "Effect": "Allow",
        "Action": [
            "ec2:CreateNetworkInterface",
            "ec2:DescribeNetworkInterfaces",
            "ec2:DetachNetworkInterface",
            "ec2:DeleteNetworkInterface"
        ],
        "Resource": "*"
    }, {
        "Sid": "Manage Route 53 records",
        "Effect": "Allow",
        "Action": [
            "route53:ChangeResourceRecordSets",
            "route53:ListResourceRecordSets"
        ],

        "Resource": ["*"]
    }]
}

To restrict the Lambda function access, you can control the scope of changes to Route 53 by specifying the hosted zones that are being managed in the format "arn:aws:route53:::hostedzone/Z148QEXAMPLE8V". This policy can be updated later if additional hosted zone IDs are added.

mirror-dns-trust.json

The mirror-dns-trust.json file contains the trust policy that grants the Lambda service permission to assume the role; this is standard for creating Lambda functions.

{
    "Version": "2012-10-17",
    "Statement": [{
        "Sid": "",
        "Effect": "Allow",
        "Principal": {
            "Service": "lambda.amazonaws.com"
        },
        "Action": "sts:AssumeRole"
    }]
}

Create IAM entities

The next step is to create the following IAM entities for the Lambda function:

• IAM policy

  Create the IAM policy using the policy document in the mirror-dns-policy.json file, replacing with the local path to the file. The output of the create-policy command includes the Amazon Resource Locator (ARN). Save the ARN, as you need it for future steps.

  aws iam create-policy \
  --policy-name mirror-dns-lambda-policy \
  --policy-document file://<LOCAL PATH>/mirror-dns-policy.json

• IAM role

  Create the IAM role using the trust policy in the mirror-dns-trust.json file, replacing with the local path to the file. The output of the create-role command includes the ARN associated with the role that you created. Save this ARN, as you need it when you create the Lambda function in the next section.

  aws iam create-role \
  --role-name mirror-dns-lambda-role \
  --assume-role-policy-document file://<LOCAL PATH>/mirror-dns-trust.json

  Attach the policy to the role. Use the ARN returned when you created the IAM policy for the
  –policy-arn input parameter.

  aws iam attach-role-policy \
  --role-name mirror-dns-lambda-role \
  --policy-arn <enter-your-policy-arn-here>

Step 3 – Create the Lambda function

The Lambda function uses modules included in the Python 2.7 Standard Library and the AWS SDK for Python module (boto3), which is preinstalled as part of the Lambda service. Additionally, the function uses the dnspython module, which provides DNS handling functions, and there is also an externalized lookup function.

The additional libraries and functions require that we create a deployment package for this example as follows:

1. Create a new directory for the Lambda function and download the Python scripts lambdafunction.py and lookuprdtype.py from the aws-lambda-ddns-function AWS Labs GitHub repo. Alternatively, clone the repo locally.

2. Install the additional dnspython module locally using the pip command. This creates a copy of the require module local to the function.

   pip install dnspython -t .

3. Update the lambda_function.py to specify proxy server configuration, if required.

4. Create a Lambda deployment package using the following command:

   zip -rq mirror-dns-lambda.zip lambda_function.py \
   lookup_rdtype.py dns*

Then, you'll use the AWS CLI to create the Lambda function and upload the deployment package by executing the following command to create the function. Note that you need to update the commands to use the ARN of the IAM role that you created earlier, as well as the local path to the Lambda deployment file containing the Python code for the Lambda function.

aws lambda create-function --function-name mirror-dns-lambda \
--runtime python2.7 \
--role <enter-your-role-arn-here> \
--handler lambda_function.lambda_handler \
--timeout 60 \
--vpc-config SubnetIds=comma-separated-vpc-subnet-ids,SecurityGroupIds=comma-separated-security-group-ids \
--memory-size 128 \
--description "DNS Mirror Function"
--zip-file fileb://<LOCAL PATH>/mirror-dns-lambda.zip

The output of the command returns the FunctionArn of the newly-created function. Save this ARN, as you need it in the next section.

Configure a test event in order to validate that your Lambda function works; it should be in JSON format similar to the following. All keys are required as well as values for Domain, MasterDns, and ZoneId.

{
    "Domain": "mydomain.com",
    "MasterDns": "10.0.0.1",
    "ZoneId": "AA11BB22CC33DD",
    "IgnoreTTL": "False",
    "ZoneSerial": ""
}

Invoke the Lambda function to test that everything is working; after the function has been invoked, check the file named output to see if the function has worked (you should see a 200 return code). Alternatively, you can test in the AWS console, using the test event to see the log output.

aws lambda invoke \
--function-name mirror-dns-lambda \
--payload fileb://event.json output

Congratulations, you've now created a secondary mirrored DNS accessible to your VPC without the need for any servers!

Step 4 – Create the CloudWatch Events rule

After you've confirmed that the Lambda function is executing correctly, you can create the CloudWatch Events rule that triggers the Lambda function on a scheduled basis. First, create a new rule with a unique name and schedule expression. You can create rules that self-trigger on schedule in CloudWatch Events, using cron or rate expressions. The following example uses a rate expression to run every 15 minutes.

aws events put-rule \
--name mirror-dns-lambda-rule \
--schedule-expression 'rate(15 minutes)'

The output of the command returns the ARN to the newly-created CloudWatch Events rule. Save the ARN, as you need it to associate the rule with the Lambda function and to set the appropriate Lambda permissions.

Next, add the permissions required for the CloudWatch Events rule to execute the Lambda function. Note that you need to provide a unique value for the –statement-id input parameter. You also need to provide the ARN of the CloudWatch Events rule that you created.

aws lambda add-permission \
--function-name mirror-dns-lambda \
--statement-id Scheduled01 \
--action 'lambda:InvokeFunction' \
--principal events.amazonaws.com \
--source-arn <enter-your-cloudwatch-events-rule-arn-here>

Finally, set the target of the rule to be the Lambda function. Because you are going to pass parameters via a JSON string, the value for –targets also needs to be in JSON format. You need to construct a file containing a unique identifier for the target, the ARN of the Lambda function previously created, and the constant text that contains the function parameters. An example targets.json file would look similar to the following; note that every quote(") in the Input value must be escaped.

[{
        "Id": "RuleStatementId01",
        "Arn": "<arn-of-lambda-function>",
        "Input": "{\"Domain\": \"mydomain.com\",\"MasterDns\": \"10.0.0.1\",\"ZoneId\": \"AA11BB22CC33DD\",\"IgnoreTTL\": \"False\",\"ZoneSerial\": \"\"}"
}]

Activate the scheduled event by adding the following target:

aws events put-targets \
--rule mirror-dns-lambda-rule \
--targets file://targets.json

Because a single rule can have multiple targets, every domain that you want to mirror can be defined as another target with a different set of parameters; change the ID and Input values in the target JSON file.

Conclusion

Now that you've seen how you can combine various AWS services to automate the mirroring of DNS to Amazon Route 53 hosted zones, we hope that you are inspired to create your own solutions using Lambda in a VPC to enable hybrid integration. Lambda allows you to create highly scalable serverless infrastructures that allow you to reduce cost and operational complexity, while providing high availability. Coupled with CloudWatch Events, you can respond to events in real-time, such as when an instance changes its state or when a customer event is pushed to the CloudWatch Events service.

To learn more about Lambda and serverless infrastructures, see the AWS Lambda Developer Guide and the " Microservices without the Servers" blog post. To learn more about CloudWatch Events, see Using CloudWatch Events in the Amazon CloudWatch Developer Guide.

We've open-sourced the code used in this example in the aws-lambda-ddns-function AWS Labs GitHub repo and can't wait to see your feedback and your ideas about how to improve the solution.

Today, we are pleased to announce the official AWS Serverless Chatbot competition!

Bots on Slack can help your team be more productive and accomplish more tasks. They can help you increase visibility into your operations or help your customers easily get information through a natural, conversational interface. However, building and running bots can be a time-consuming and difficult task. Developers must provision, manage, and scale the compute resources that run the bot code.

With AWS Lambda, it’s easy to build and run bots. Upload your code and Lambda takes care of everything required to run and scale your code with high availability.

Enter your bot for a chance to win a ticket to re:Invent 2016 in Las Vegas!

Enter the competition by building a Slack bot that runs on AWS Lambda and Amazon API Gateway. You’re also encouraged to integrate Slack APIs or other APIs, SDKs, and datasets so long as you are authorized to use them.

We’ll be selecting 8 winners for a prize. Each prize includes one ticket to AWS re:Invent and access to discounted hotel room rates, along with recognition at the Serverless State of the Union address, some cool swag, $100 in AWS credits, and publicity opportunities for the winning bots.

To help you get started, we’ve created a few very basic bots that take advantage of Slack’s triggers and webhooks. These bots are built using AWS Lambda and Amazon API Gateway. To get started, check out the diagram below and view the code samples and instructions on GitHub at: aws-serverless-chatbot-sample.

Here’s what you need to do to enter:

1. Read the Rules and Eligibility Guidelines.
2. Register for the AWS Serverless Chatbot Competition.
3. Create AWS and Slack developer accounts.
4. Visit the Resources Page to learn more about the APIs and services.
5. Build your chatbot. Our sample code (aws-serverless-chatbot-sample) is a good place to start.
6. Create your demo video and other materials for the submission.
7. Submit your materials before 5 PM ET on September 29, 2016.