AWS Compute Blog
Node.js 4.3.2 Runtime Now Available on Lambda
We are happy to announce that you may now develop your AWS Lambda functions using the Node.js 4.3.2 runtime. You can start using this new runtime version today by specifying a runtime parameter value of “nodejs4.3” when creating or updating functions. We will continue to support creating new Lambda functions on Node.js 0.10. However starting October 2016 you will no longer be able to create functions using Node.js 0.10, given the upcoming end of life for the runtime. Here’s a quick primer on what’s changed between the two versions:
New Node features
You can now leverage features in the V8 Javascript Engine such as ES6 Support, block scoping, Promises, and new arrow functions, to name a few. For more information, see the Expressive ES6 features that shine in Node.js 4.0 post by Ryan Paul @ RethinkDB.
Backward compatible
Nothing in regards to your existing functions running under Node.js 0.10 will change, and they will continue to operate and function as expected. You may also port your existing Node.js 0.10 functions over to Node.js 4.3.2 by simply updating the runtime, and they will continue to work as written. You will however need to take into account any static modules you may have compiled for 0.10 before making this move. Be sure to review the API changes between Node.js 0.10 and Node.js 4 to see if there are other changes that affect your code.
Node callbacks
The programming model for Node.js 0.10, Lambda required an explicit Context method call (done(), suceeed(), fail()) to exit the function. Context.succeed, context.done, and context.fail however, are more than just bookkeeping – they cause the request to return after the current task completes and freeze the process immediately, even if other tasks remain in the Node.js event loop. Generally that’s not what you want if those tasks represent incomplete callbacks.
This programming model for Node.js 4.3.2 improves on this by adding an optional callback parameter to the method. The callback parameter can be used to specify error or return values for the function execution. You specify the optional callback parameter when defining your function handler as below:
exports.myHandler = (event, context, callback) => callback(null, "I'm running Node4!");
By default, the callback waits for all the tasks in the Node.js event loop to complete, just as it would if you ran the function locally. If you chose to not use the callback parameter in your code, then AWS Lambda implicitly calls it with a return value of null. You can still use the Context methods to terminate the function, but the callback approach of waiting for all tasks to complete is more idiomatic to how Node.js behaves in general. The context parameter will also continue to exist and provides your handler with the runtime information of the Lambda function that is executing.
If you want to simulate the same behavior as a context method, you now have the ability to access the callbackWaitsForEmptyEventLoop setting via the context object. This property is useful to modify the default behavior of the callback. You can set this property to false to request AWS Lambda to freeze the process after the callback is called. For more information about this new functionality, see Lambda Function Handler.
Remember, the existing Node.js 0.10 programming model does not support the new callback functionality that specifically exists in the new 4.3.2 runtime. If you continue to use 0.10, you will still need to take advantage of the context object to specify return values of your function.
For more information, see the AWS Lambda Developer Guide.
Hope you enjoy,
-Bryan
Have feedback? I’m always listening @listonb
Amazon API Gateway mapping improvements
Yesterday we announced the new Swagger import API. You may have also noticed a new first time user experience in the API Gateway console that automatically creates a sample Pet Store API and guides you though API Gateway features. That is not all we’ve been doing:
Over the past few weeks, we’ve made mapping requests and responses easier. This post takes you through the new features we introduced and gives practical examples of how to use them.
Multiple 2xx responses
We heard from many of you that you want to return more than one 2xx response code from your API. You can now configure Amazon API Gateway to return multiple 2xx response codes, each with its own header and body mapping templates. For example, when creating resources, you can return 201 for “created” and 202 for “accepted”.
Context variables in parameter mapping
We have added the ability to reference context variables from the parameter mapping fields. For example, you can include the identity principal or the stage name from the context variable in a header to your HTTP backend. To send the principalId returned by a custom authorizer in an X-User-ID header to your HTTP backend, use this mapping expression:
context.authorizer.principalId
For more information, see the context variable in the Mapping Template Reference page of the documentation.
Access to raw request body
Mapping templates in API Gateway help you transform incoming requests and outgoing responses from your API’s back end. The $input variable in mapping templates enables you to read values from a JSON body and its properties. You can now also return the raw payload, whether it’s JSON, XML, or a string using the $input.body property.
For example, if you have configured your API to receive raw data and pass it to Amazon Kinesis using an AWS service proxy integration, you can use the body property to read the incoming body and the $util variable to encode it for an Amazon Kinesis stream.
{
"Data" : "$util.base64Encode($input.body)",
"PartitionKey" : "key",
"StreamName" : "Stream"
}
JSON parse function
We have also added a parseJson() method to the $util object in mapping templates. The parseJson() method parses stringified JSON input into its object representation. You can manipulate this object representation in the mapping templates. For example, if you need to return an error from AWS Lambda, you can now return it like this:
exports.handler = function(event, context) {
var myErrorObj = {
errorType : "InternalFailure",
errorCode : 9130,
detailedMessage : "This is my error message",
stackTrace : ["foo1", "foo2", "foo3"],
data : {
numbers : [1, 2, 3]
}
}
context.fail(JSON.stringify(myErrorObj));
};
Then, you can use the parseJson() method in the mapping template to extract values from the error and return a meaningful message from your API, like this:
#set ($errorMessageObj = $util.parseJson($input.path('$.errorMessage')))
#set ($bodyObj = $util.parseJson($input.body))
{
"type" : "$errorMessageObj.errorType",
"code" : $errorMessageObj.errorCode,
"message" : "$errorMessageObj.detailedMessage",
"someData" : "$errorMessageObj.data.numbers[2]"
}
This will produce a response that looks like this:
{
"type" : "InternalFailure",
"code" : 9130,
"message" : "This is my error message",
"someData" : "3"
}
Conclusion
We continuously release new features and improvements to Amazon API Gateway. Your feedback is extremely important and guides our priorities. Keep sending us feedback on the API Gateway forum and on social media.
Indexing Amazon DynamoDB Content with Amazon Elasticsearch Service Using AWS Lambda
Stephan Hadinger |
Mathieu Cadet Account Representative |
A lot of AWS customers have adopted Amazon DynamoDB for its predictable performance and seamless scalability. The main querying capabilities of DynamoDB are centered around lookups using a primary key. However, there are certain times where richer querying capabilities are required. Indexing the content of your DynamoDB tables with a search engine such as Elasticsearch would allow for full-text search.
In this post, we show how you can send changes to the content of your DynamoDB tables to an Amazon Elasticsearch Service (Amazon ES) cluster for indexing, using the DynamoDB Streams feature combined with AWS Lambda.
Architectural overview
Here’s a high-level overview of the architecture:
We’ll cover the main steps required to put this bridge in place:
- Choosing the DynamoDB tables to index and enabling DynamoDB Streams on them.
- Creating an IAM role for accessing the Amazon ES cluster.
- Configuring and enabling the Lambda blueprint.
Choosing the DynamoDB table to index
In this post, you look at indexing the content of a product catalog in order to provide full-text search capabilities. You’ll index the content of a DynamoDB table called all_products, which is acting as the catalog of all products.
Here’s an example of an item stored in that table:
{
"product_id": "B016JOMAEE",
"name": "Serverless Single Page Apps: Fast, Scalable, and Available",
"category": "ebook",
"description": "AWS Lambda - A Guide to Serverless Microservices
takes a comprehensive look at developing
serverless workloads using the new
Amazon Web Services Lambda service.",
"author": "Matthew Fuller",
"price": 15.0,
"rating": 4.8
}Enabling DynamoDB Streams
In the DynamoDB console, enable the DynamoDB Streams functionality on the all_products table by selecting the table and choosing Manage Stream.
Multiple options are available for the stream. For this use case, you need new items to appear in the stream; choose either New image or New and old images. For more information, see Capturing Table Activity with DynamoDB Streams.
After the stream is set up, make a good note of the stream ARN. You’ll need that information later, when configuring the access permissions.
Creating a new IAM role
The Lambda function needs read access to the DynamoDB stream just created. In addition, the function also requires access to the Amazon ES cluster to submit new records for indexing.
In the AWS Identity and Access Management (IAM) console, create a new role for the Lambda function and call it ddb-elasticsearch-bridge.
As this role will be used by the Lambda function, choose AWS Lambda from the AWS Service Roles list.
On the following screens, choose the AWSLambdaBasicExecutionRole managed policy, which allows the Lambda function to send logs to Amazon CloudWatch Logs.
Configuring access to the Amazon ES cluster
First, you need a running Amazon ES cluster. In this example, create a search domain called inventory. After the domain has been created, note its ARN:
In the IAM console, select the ddb-elasticsearch-bridge role created earlier and add two inline policies to that role:
Here’s the policy to add to allow the Lambda code to push new documents to Amazon ES (replace the resource ARN with the ARN of your Amazon ES cluster):
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"es:ESHttpPost"
],
"Effect": "Allow",
"Resource": "arn:aws:es:us-east-1:0123456789:domain/inventory/*"
}
]
}Important: you need to add /* to the resource ARN as depicted above.
Next, add a second policy for read access to the DynamoDB stream (replace the resource ARN with the ARN of your DynamoDB stream):
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"dynamodb:DescribeStream",
"dynamodb:GetRecords",
"dynamodb:GetShardIterator",
"dynamodb:ListStreams"
],
"Effect": "Allow",
"Resource": [
"arn:aws:dynamodb:us-east-1:0123456789:table/all_products/stream/2016-02-16T23:13:07.600"
]
}
]
}Enabling the Lambda blueprint
When you log into the Lambda console and choose Create a Lambda Function, you are presented with a list of blueprints to use. Select the blueprint called dynamodb-to-elasticsearch.
Next, select the DynamoDB table all_products as the event source:
Then, customize the Lambda code to specify the Elasticsearch endpoint:
Finally, select the ddb-elasticsearch-bridge role created earlier to give the Lambda function the permissions required to interact with DynamoDB and the Amazon ES cluster:
Testing the result
You’re all set!
After a few records have been added to your DynamoDB table, you can go back to the Amazon ES console and validate that a new index for your items has been automatically created:
Playing with Kibana (Optional)
Elasticsearch is commonly used with Kibana for visual exploration of data.
To start querying the indexed data, create an index pattern in Kibana. Use the name of the DynamoDB table as an index pattern:
Kibana automatically determines the best type for each field:
Use a simple query to search the product catalog for all items in the category book containing the word aws in any field:
Other considerations
Indexing pre-existing content
The solution presented earlier is ideal to ensure that new data is indexed as soon it is added to the DynamoDB table. But what about pre-existing data stored in the table?
Luckily, the Lambda function used earlier can also be used to process data from an Amazon Kinesis stream, as long as the format of the data is similar to the DynamoDB Streams records.
Provided that you have an Amazon Kinesis stream set up as an additional input source for the Lambda code above, you can use the (very naive) sample Python3 code below to read the entire content of a DynamoDB table and push it to an Amazon Kinesis stream called ddb-all-products for indexing in Amazon ES.
import json
import boto3
import boto3.dynamodb.types
# Load the service resources in the desired region.
# Note: AWS credentials should be passed as environment variables
# or through IAM roles.
dynamodb = boto3.resource('dynamodb', region_name="us-east-1")
kinesis = boto3.client('kinesis', region_name="us-east-1")
# Load the DynamoDB table.
ddb_table_name = "all_products"
ks_stream_name = "ddb-all-products"
table = dynamodb.Table(ddb_table_name)
# Get the primary keys.
ddb_keys_name = [a['AttributeName'] for a in table.attribute_definitions]
# Scan operations are limited to 1 MB at a time.
# Iterate until all records have been scanned.
response = None
while True:
if not response:
# Scan from the start.
response = table.scan()
else:
# Scan from where you stopped previously.
response = table.scan(ExclusiveStartKey=response['LastEvaluatedKey'])
for i in response["Items"]:
# Get a dict of primary key(s).
ddb_keys = {k: i[k] for k in i if k in ddb_keys_name}
# Serialize Python Dictionaries into DynamoDB notation.
ddb_data = boto3.dynamodb.types.TypeSerializer().serialize(i)["M"]
ddb_keys = boto3.dynamodb.types.TypeSerializer().serialize(ddb_keys)["M"]
# The record must contain "Keys" and "NewImage" attributes to be similar
# to a DynamoDB Streams record. Additionally, you inject the name of
# the source DynamoDB table in the record so you can use it as an index
# for Amazon ES.
record = {"Keys": ddb_keys, "NewImage": ddb_data, "SourceTable": ddb_table_name}
# Convert the record to JSON.
record = json.dumps(record)
# Push the record to Amazon Kinesis.
res = kinesis.put_record(
StreamName=ks_stream_name,
Data=record,
PartitionKey=i["product_id"])
print(res)
# Stop the loop if no additional records are
# available.
if 'LastEvaluatedKey' not in response:
breakNote: In the code example above, you are passing the name of the source DynamoDB table as an extra record attribute SourceTable. The Lambda function uses that attribute to build the Amazon ES index name. Another approach for passing that information is tagging the Amazon Kinesis stream.
Now, create the Amazon Kinesis stream ddb-all-productsand then add permissions to the ddb-elasticsearch-bridge role in IAM to allow the Lambda function to read from the stream:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"kinesis:Get*",
"kinesis:DescribeStream"
],
"Resource": [
"arn:aws:kinesis:us-east-1:0123456789:stream/ddb-all-products"
]
}
]
}Finally, set the Amazon Kinesis stream as an additional input source to the Lambda function:
Neat tip: Doing a full re-index of the content this way will not create duplicate entries in Amazon ES.
Paying attention to attribute types
With DynamoDB, you can use different types for the same attribute on different records, but Amazon ES expects a given attribute to be of only one type. Similarly, changing the type of an existing attribute after it has been indexed in Amazon ES causes problems and some searches won’t work as expected.
In these cases, you must rebuild the Amazon ES index. For more information, see Reindexing Your Data in the Elasticsearch documentation.
Conclusion
In this post, you have seen how you can use AWS Lambda with DynamoDB to index your table content in Amazon ES as changes happen.
Because you are relying entirely on Lambda for the business logic, you don’t have to deal with servers at any point: everything is managed by the AWS platform in a highly available and scalable fashion. To learn more about Lambda and serverless infrastructures, see the Microservices without the Servers blog post.
Now that you have added full-text search to your DynamoDB table, you might be interested in exposing its content through a small REST API. For more information, see Using Amazon API Gateway as a proxy for DynamoDB.
Building a Dynamic DNS for Route 53 using CloudWatch Events and Lambda
Jeremy Cowan, AWS Solutions Architect |
Efrain Fuentes, Enterprise Solutions Architect |
Introduction
Dynamic registration of resource records is useful when you have instances that are not behind a load balancer that you would like to address by a host name and domain suffix of your choosing, rather than the default <region>.compute.internal or ec2.internal.
In this post, we explore how you can use CloudWatch Events and Lambda to create a Dynamic DNS for Route 53. Besides creating A records, this solution allows you to create alias, i.e. CNAME records, for when you want to address a server by a “friendly” or alternate name. Although this is antithetical to treating instances as disposable resources, there are still a lot of shops that find this useful.
Using CloudWatch and Lambda to respond to infrastructure changes in real-time
With the advent of CloudWatch Events in January 2016, you can now get near real-time information when an AWS resource changes its state, including when instances are launched or terminated. When you combine this with the power of Amazon Route 53 and AWS Lambda, you can create a system that closely mimics the behavior of Dynamic DNS.
For example, when a newly-launched instance changes its state from pending to running, an event can be sent to a Lambda function that creates a resource record in the appropriate Route 53 hosted zone. Similarly, when instances are stopped or terminated, Lambda can automatically remove resource records from Route 53.
The example provided in this post works precisely this way. It uses information from a CloudWatch event to gather information about the instance, such as its public and private DNS name, its public and private IP address, the VPC ID of the VPC that the instance was launched in, its tags, and so on. It then uses this information to create A, PTR, and CNAME records in the appropriate Route 53 public or private hosted zone. The solution persists data about the instances in an Amazon DynamoDB table so it can remove resource records when instances are stopped or terminated.
Route 53 Hosted zones
Route 53 offers the convenience of domain name services without having to build a globally distributed highly reliable DNS infrastructure. It allows instances within your VPC to resolve the names of resources that run within your AWS environment. It also lets clients on the Internet resolve names of your public-facing resources. This is accomplished by querying resource record sets that reside within a Route 53 public or private hosted zone.
A private hosted zone is basically a container that holds information about how you want to route traffic for a domain and its subdomains within one or more VPCs, whereas a public hosted zone is a container that holds information about how you want to route traffic from the Internet.
Choosing between VPC DNS or Route 53 Private Hosted Zones
Admittedly, you can use VPC DNS for internal name resolution instead of Route 53 private hosted zones. Although it doesn’t dynamically create resource records, VPC DNS will provide name resolution for all the hosts within a VPC’s CIDR range.
Unless you create a DHCP option set with a custom domain name and disable hostnames at the VPC, you can’t change the domain suffix; all instances are either assigned the ec2.internal or <region>.compute.internal domain suffix. You can’t create aliases or other resource record types with VPC DNS either.
Private hosted zones help you overcome these challenges by allowing you to create different resource record types with a custom domain suffix. Moreover, with Route 53 you can create a subdomain for your current DNS namespace or you can migrate an existing subdomain to Route 53. By using these options, you can create a contiguous DNS namespace between your on-premises environment and AWS.
So, while VPC DNS can provide basic name resolution for your VPC, Route 53 private hosted zones offer richer functionality by comparison. It also has a programmable API that can be used to automate the creation/removal of records sets and hosted zones which we’re going leverage later in this post.
Route 53 doesn’t offer support for dynamic registration of resource record sets for public or private hosted zones. This can pose challenges when an Auto Scaling event occurs and the instances are not behind a load balancer. A common workaround is to use an automation framework like Chef, Puppet, Ansible, or Salt to create resource records, or by adding instance user data to the launch profile of the Auto Scaling group. The drawbacks to these approaches are that:
1) automation frameworks typically require you to manage additional infrastructure.
2) instance user data doesn’t handle the removal of resource records when the instance is terminated.
This was the motivation for creating a serverless architecture that dynamically creates and removes resource records from Route 53 as EC2 instances are created and destroyed.
DDNS/Lambda example
Make sure that you have the latest version of the AWS CLI installed locally. For more information, see Getting Set Up with the AWS Command Line Interface.
For this example, create a new VPC configured with a private and public subnet, using Scenario 2: VPC with Public and Private Subnets (NAT) from the Amazon VPC User Guide. Ensure that the VPC has the DNS resolution and DNS hostnames options set to yes.
After the VPC is created, you can proceed to the next steps.
Step 1 – Create an IAM role for the Lambda function
In this step, you use the AWS Command Line Interface (AWS CLI) to create the Identity and Access Management (IAM) role that the Lambda function assumes when the function is invoked. You need to create an IAM policy with the required permissions and then attach this policy to the role.
Download the ddns-policy.json and ddns-trust.json files from the AWS Labs GitHub repo.
ddns-policy.json
The policy includes ec2:Describe permission, required for the function to obtain the EC2 instance’s attributes, including the private IP address, public IP address, and DNS hostname. The policy also includes DynamoDB and Route 53 full access, required for the function to create the DynamoDB table and to update the Route 53 DNS records. The policy also allows the function to create log groups and log events.
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": "ec2:Describe*",
"Resource": "*"
}, {
"Effect": "Allow",
"Action": [
"dynamodb:*"
],
"Resource": "*"
}, {
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "*"
}, {
"Effect": "Allow",
"Action": [
"route53:*"
],
"Resource": [
"*"
]
}]
}
ddns-trust.json
The ddns-trust.json file contains the trust policy that grants the Lambda service permission to assume the role.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "lambda.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
Create the policy using the policy document in the ddns-pol.json file. You need to replace <LOCAL PATH> with your local path to the ddns-pol.json file. The output of the aws iam create-policy command includes the Amazon Resource Locator (ARN). Save the ARN, since you will need it for future steps.
aws iam create-policy --policy-name ddns-lambda-policy --policy-document file://<LOCAL PATH>/ddns-pol.json
Create the ddns-lambda-role IAM role using the trust policy in the ddns-trust.json file. You need to replace <LOCAL PATH> with your local path to the ddns-trust.json file. The output of the aws iam create-role command includes the ARN associated with the role that you created. Save this ARN, since you will need it when you create the Lambda function in the next section.
aws iam create-role --role-name ddns-lambda-role --assume-role-policy-document file://<LOCAL PATH>/ddns-trust.json
Attach the policy to the role. Use the ARN returned in step 2 for the –policy-arn input parameter.
aws iam attach-role-policy --role-name ddns-lambda-role --policy-arn <enter-your-policy-arn-here>
Step 2 – Create the Lambda function
The Lambda function uses modules included in the Python 2.7 Standard Library and the AWS SDK for Python module (boto3), which is preinstalled as part of the Lambda service. As such, you do not need to create a deployment package for this example.
The function code performs the following:
- Checks to see whether the “DDNS” table exists in DynamoDB and creates the table if it does not. This table is used to keep a record of instances that have been created along with their attributes. It’s necessary to persist the instance attributes in a table because once an EC2 instance is terminated, its attributes are no longer available to be queried via the EC2 API. Instead, they must be fetched from the table.
- Queries the event data to determine the instance’s state. If the state is “running”, the function queries the EC2 API for the data it will need to update DNS. If the state is anything else, e.g. “stopped” or “terminated”, it will retrieve the necessary information from the “DDNS” DynamoDB table.
- Verifies that “DNS resolution” and “DNS hostnames” are enabled for the VPC, as these are required in order to use Route 53 for private name resolution. The function then checks whether a reverse lookup zone for the instance already exists. If it does, it checks to see whether the reverse lookup zone is associated with the instance’s VPC. If it isn’t, it creates the association. This association is necessary in order for the VPC to use Route 53 zone for private name resolution.
- Checks the EC2 instance’s tags for the CNAME and ZONE tags. If the ZONE tag is found, the function creates A and PTR records in the specified zone. If the CNAME tag is found, the function creates a CNAME record in the specified zone.
- Verifies whether there’s a DHCP option set assigned to the VPC. If there is, it uses the value of the domain name to create resource records in the appropriate Route 53 private hosted zone. The function also checks to see whether there’s an association between the instance’s VPC and the private hosted zone. If there isn’t, it creates it.
- Deletes the required DNS resource records if the state of the EC2 instance changes to “shutting down” or “stopped”.
Use the AWS CLI to create the Lambda function:
- Download the union.py.zip file from the AWS Labs GitHub repo.
- Execute the following command to create the function. Note that you need to update the command to use the ARN of the role that you created earlier, as well as the local path to the union.py.zip file containing the Python code for the Lambda function.
aws lambda create-function --function-name ddns_lambda --runtime python2.7 --role <enter-your-role-arn-here> --handler union.lambda_handler --timeout 30 --zip-file fileb://<LOCAL PATH>/union.py.zip
- The output of the command returns the ARN of the newly-created function. Save this ARN, as you will need it in the next section
Step 3 – Create the CloudWatch Events Rule
In this step, you create the CloudWatch Events rule that triggers the Lambda function whenever CloudWatch detects a change to the state of an EC2 instance. You configure the rule to fire when any EC2 instance state changes to “running”, “shutting down”, or “stopped”. Use the aws events put-rule command to create the rule and set the Lambda function as the execution target:
aws events put-rule --event-pattern "{\"source\":[\"aws.ec2\"],\"detail-type\":[\"EC2 Instance State-change Notification\"],\"detail\":{\"state\":[\"running\",\"shutting-down\",\"stopped\"]}}" --state ENABLED --name ec2_lambda_ddns_rule
The output of the command returns the ARN to the newly created CloudWatch Events rule, named ec2_lambda_ddns_rule. Save the ARN, as you will need it to associate the rule with the Lambda function and to set the appropriate Lambda permissions.
Next, set the target of the rule to be the Lambda function. Note that the —targets input parameter requires that you include a unique identifier for the Id target. You also need to update the command to use the ARN of the Lambda function that you created previously.
aws events put-targets --rule ec2_lambda_ddns_rule --targets Id=id123456789012,Arn=<enter-your-lambda-function-arn-here>
Next, you add the permissions required for the CloudWatch Events rule to execute the Lambda function. Note that you need to provide a unique value for the –statement-id input parameter. You also need to provide the ARN of the CloudWatch Events rule you created earlier.
aws lambda add-permission --function-name ddns_lambda --statement-id 45 --action lambda:InvokeFunction --principal events.amazonaws.com --source-arn <enter-your-cloudwatch-events-rule-arn-here>
Step 4 – Create the private hosted zone in Route 53
To create the private hosted zone in Route 53, follow the steps outlined in Creating a Private Hosted Zone.
Step 5 – Create a DHCP options set and associate it with the VPC
In this step, you create a new DHCP options set and set the domain to be that of your private hosted zone.
- Follow the steps outlined in Creating a DHCP Options Set to create a new set of DHCP options.
- In the Create DHCP options set dialog box, give the new options set a name, set Domain name to the name of the private hosted zone that you created in Route 53, and set Domain name servers to “AmazonProvidedDNS”. Choose Yes, Create.
- Next, follow the steps outlined in Changing the Set of DHCP Options a VPC Uses to update the VPC to use the newly-created DHCP options set.
Step 6 – Launching the EC2 instance and validating results
In this step, you launch an EC2 instance and verify that the function executed successfully.
As mentioned previously, the Lambda function looks for the ZONE or CNAME tags associated with the EC2 instance. If you specify these tags when you launch the instance, you should include the trailing dot. In this example, the ZONE tag would be set to “ddnslambda.com.” and the CNAME tag could be set to “test.ddnslambda.com.”.
Because you updated the DHCP options set in this example, the Lambda function uses the specified zone when it creates the Route 53 DNS resource records. You can use the ZONE tag to override this behavior if you want the function to update a different hosted zone.
In this example, you launch an EC2 instance into the private subnet of the VPC. Because you updated the domain value of the DHCP options set to be that of the private hosted zone, the Lambda function creates the DNS resource records in the Route 53 zone file.
Launching the EC2 instance
- Follow the steps to launch an EC2 instance outlined in Launching an Instance.
- In Step 3: Configure Instance Details, for Network, select the VPC. For Subnet, select the private subnet. Choose Review and Launch.
- (Optional) If you would like to update a different private hosted zone than the one you associated with the VPC, specify the ZONE tag in this step. You can also specify the CNAME tag if you would like the function to create a CNAME resource record in the associated zone.
Choose Edit tags in the Step 7: Review Instance Launch.
Enter the key and value for Step 5: Tag Instance then choose Review and Launch.
Complete the launch of the instance and wait until the instance state changes to “running”. Then, continue to the next step.
Validating results
In this step, you verify that your Lambda function successfully updated the Route 53 resource records.
- Log in to the Route 53 console.
- In the left navigation pane, choose Hosted Zones to view the list of private and public zones currently configured in Route 53.
- Select the hosted zone that you created in step 4 to view the zone file.
- Verify that the resource records were created.
- Now that you’ve verified that the Lambda function successfully updated the Route 53 resource records in the zone file, stop the EC2 instance and verify that the records are removed by the function.
- Log in to the EC2 console.
- Choose Instances in the left navigation pane.
- Select the EC2 instance you launched earlier and choose Stop.
- Follow Steps 1 – 3 to view the DNS resource records in the Route 53 zone.
- Verify that the records have been removed from the zone file by the Lambda function.
Conclusion
Now that you’ve seen how you can combine various AWS services to automate the creation and removal of Route 53 resource records, we hope you are inspired to create your own solutions. CloudWatch Events is a powerful tool because it allows you to respond to events in real-time, such as when an instance changes its state. When used with Lambda, you can create highly scalable serverless infrastructures that react instantly to infrastructure changes.
To learn more about CloudWatch Events, see Using CloudWatch Events in the Amazon CloudWatch Developer Guide. To learn more about Lambda and serverless infrastructures, see the AWS Lambda Developer Guide and the “Microservices without the Servers” blog post.
We’ve open-sourced the code in this example in the AWS Labs GitHub repo and can’t wait to see your feedback and your ideas about how to improve the solution
Cloudmicro for AWS: Speeding up serverless development at The Coca‑Cola Company
We have a guest blog post today from our friend Patrick Brandt at The Coca‑Cola Company. Patrick and his team have open-sourced an innovative use of Docker containers to encourage rapid local development and testing for applications that use AWS Lambda and Amazon DynamoDB.
Using Cloudmicro to build AWS Lambda and DynamoDB applications on your laptop
My team at The Coca‑Cola Company recently began work on a proximity-marketing platform using AWS Lambda and DynamoDB. We’re gathering beacon sighting events via API Gateway, layering in additional data with a Lambda function, and then storing these events in DynamoDB.
In an effort to shorten the development cycle-time of building and deploying Lambda functions, we created a local runtime of Lambda and DynamoDB using Docker containers. Running our Lambda functions locally in containers removed the overhead of having to deploy code to debug it, greatly increasing the speed at which we could build and tweak new features. I’ve since launched an open-source organization called Cloudmicro with the mission of assembling Docker-ized versions of AWS services to encourage rapid development and easy experimentation.
Getting started with Cloudmicro
The Cloudmicro project I’m working with is a local runtime for Python-based Lambda functions that integrate with DynamoDB: https://github.com/Cloudmicro/lambda-dynamodb-local. The only prerequisite for this project is that you have Docker installed and running on your local environment.
Cloning the lambda-dynamodb-local project and running the hello function
In these examples, you run commands using Docker on a Mac. The instructions for running Docker commands using Windows are slightly different and can be found in the project Readme.
Run the following commands in your terminal window to clone the lambda-dynamodb-local project and execute the hello Lambda function:
> git clone https://github.com/Cloudmicro/lambda-dynamodb-local.git > cd lambda-dynamodb-local > docker-compose up -d > docker-compose run --rm -e FUNCTION_NAME=hello lambda-python
Your output will look like this:
executing hello function locally:
[root - INFO - 2016-02-29 14:55:30,382] Event: {u'first_name': u'Umberto', u'last_name': u'Boccioni'}
[root - INFO - 2016-02-29 14:55:30,382] START RequestId: 11a94c54-d0fe-4a87-83de-661692edc440
[root - INFO - 2016-02-29 14:55:30,382] END RequestId: 11a94c54-d0fe-4a87-83de-661692edc440
[root - INFO - 2016-02-29 14:55:30,382] RESULT:
{'message': 'Hello Umberto Boccioni!'}
[root - INFO - 2016-02-29 14:55:30,382] REPORT RequestId: 11a94c54-d0fe-4a87-83de-661692edc440 Duration: 0.11 ms
The output is identical to what you would see if you had run this same function using AWS.
Understanding how the hello function runs locally
We’ll look at three files and the docker-compose command to understand how the hello function executes with its test event.
The docker-compose.yml file
The docker-compose.yml file defines three docker-compose services:
lambda-python: build: . container_name: python-lambda-local volumes: - ./:/usr/src links: - dynamodb working_dir: /usr/src dynamodb: container_name: dynamodb-local image: modli/dynamodb expose: - "8000" init: image: node:latest container_name: init-local environment: - DYNAMODB_ENDPOINT=http://dynamodb:8000 volumes: - ./db_gen:/db_gen links: - dynamodb working_dir: /db_gen command: /bin/bash init.sh
- lambda-python contains the local version of the Python-based Lambda runtime that executes the Lambda function handler in lambda_functions/hello/hello.py.
- dynamodb contains an instance of the dynamodb-local application (a fully functional version of DynamoDB).
- init contains an application that initializes the dynamodb service with any number of DynamoDB tables and optional sample data for those tables.
The hello function only uses the lambda-python service. You’ll look at an example that uses dynamodb and init a little later.
The lambda_functions/hello/hello.py file
The hello function code is identical to the Lambda code found in the AWS documentation for Python handler functions:
import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
def hello_handler(event, context):
message = 'Hello {} {}!'.format(event['first_name'],
event['last_name'])
return {
'message' : message
}
Like the hello function, your Lambda functions will live in a subdirectory of lambda_functions. The pattern you’ll follow is lambda_functions/{function name}/{function name}.py and the function handler in your Python file will be named {function name}_handler.
You can also include a requirements.txt file in your function directory that will include any external Python library dependencies required by your Lambda function.
The local_events/hello.json file
The test event for the hello function has two fields:
{
"first_name": "Umberto",
"last_name": "Boccioni"
}
All test events live in the local_events directory. By convention, the file names for each test event must match the name of the corresponding Lambda function in the lambda_functions directory.
The docker-compose command
Running the docker-compose command will instantiate containers for all of the services outlined in the docker-compose.yml file and execute the hello function.
docker-compose run --rm -e FUNCTION_NAME=hello lambda-python
- The docker-compose run command will bring up the lambda-python service and the dynamodb linked service defined in the docker-compose.yml file.
- The –rm argument instructs docker-compose to destroy the container running the Lambda function once the function is complete.
- The -e FUNCTION_NAME=hello argument defines an environment variable that the Lambda function container uses to run a specific function in the lambda_functions directory (-e FUNCTION_NAME=hello will run the hello function).
Using DynamoDB
Now we’ll look at how you use the init service to create DynamoDB tables and seed them with sample data. Then we’ll tie it all together and create a Lambda function that reads data from a table in the DynamoDB container.
Creating tables and populating them with data
The init service uses two subdirectories in the db_gen directory to set up the tables in the container created by the dynamodb service:
- db_gen/tables/ contains JSON files that define each DynamoDB table.
- db_gen/table_data/ contains optional JSON files that define a list of items to be inserted into each table.
The file names in db_gen/table_data/ must match those in db_gen/tables/ in order to load tables with data.
You’ll need to follow a couple of steps to allow the init service to automatically create your DynamoDB tables and populate them with sample data. In this example, you’ll be creating a table that stores English words.
- Add a file named “words.json” to db_gen/tables.
{ "AttributeDefinitions": [ { "AttributeName": "language_code", "AttributeType": "S" }, { "AttributeName": "word", "AttributeType": "S" } ], "GlobalSecondaryIndexes": [ { "IndexName": "language_code-index", "Projection": { "ProjectionType": "ALL" }, "ProvisionedThroughput": { "WriteCapacityUnits": 5, "ReadCapacityUnits": 5 }, "KeySchema": [ { "KeyType": "HASH", "AttributeName": "language_code" } ] } ], "ProvisionedThroughput": { "WriteCapacityUnits": 5, "ReadCapacityUnits": 5 }, "TableName": "words", "KeySchema": [ { "KeyType": "HASH", "AttributeName": "word" } ] } - Add a file named “words.json” to db_gen/table_data.
[{"word":"a","langauge_code":"en"}, {"word":"aah","langauge_code":"en"}, {"word":"aahed","langauge_code":"en"}, {"word":"aahing","langauge_code":"en"}, {"word":"aahs","langauge_code":"en"}]
Your DynamoDB database can be re-created with the init service by running this command:
docker-compose run --rm init
This will rebuild your DynamoDB container with your table definitions and table data.
You can use the describe-table command in the AWS CLI as a handy way to create your DynamoDB table definitions: first use the AWS console to create a DynamoDB table within your AWS account and then use the describe-table command to return a JSON representation of that table. If you use this shortcut, be aware that you’ll need to massage the CLI response such that the “Table” field is removed and the JSON in the “Table” field is moved up a level in the hierarchy. Once this is done, there are several fields you need to remove from the response before it can be used to create your DynamoDB table. You can use the validation errors returned by running the following command as your guide for the fields that need to be removed:
docker-compose run --rm init
Retrieving data from DynamoDB
Now you’re going to write a Lambda function that scans the words table in DynamoDB and returns the output.
- Create a getWords Lambda function in lambda_functions/getWords/getWords.py.
from lambda_utils import * import logging logger = logging.getLogger() logger.setLevel(logging.INFO) @import_config def getWords_handler(event, context, config): dynamodb = dynamodb_connect(config) words_table = dynamodb.Table(config.Dynamodb.wordsTable) words = words_table.scan() return words["Items"]
- Create local_events/getWords.json and add an empty JSON object.
{} - Ensure that the table name is referenced in config/docker-config.py.
class Dynamodb: wordsTable = "words" endpoint = "http://dynamodb:8000" class Session: region = "us-east-1" access_key = "Temp" secret_key = "Temp"
- Now you can run your new function and see the results of a word table scan.
docker-compose run --rm -e FUNCTION_NAME=getWords lambda-python
You may have noticed the @import_config decorator applied to the Lambda function handler in the prior example. This is a utility that imports configuration information from the config directory and injects it into the function handler parameter list. You should update the config/docker-config.py file with DynamoDB table names and then reference these table names via the config parameter in your Lambda function handler.
This configuration pattern is not specific to Lambda functions run with Cloudmicro; it is an example of a general approach to environmental-awareness in Python-based Lambda that I’ve outlined on Gist.
Call for contributors
The goal of Cloudmicro for AWS is to re-create the AWS cloud on your laptop for rapid development of cloud applications. The lambda-dynamodb-local project is just the start of a much larger vision for an ecosystem of interconnected Docker-ized AWS components.
Here are some milestones:
- Support Lambda function invocation from other Docker-ized Lambda functions.
- Add a Docker-ized S3 service.
- Create Yeoman generators to easily scaffold Cloudmicro services.
Supporting these capabilities will require re-architecting the current lambda-dynamodb-local project into a system that provides more robust coordination between containers. I’m hoping to enlist brilliant developers like you to support the cause and build something that many people will find useful.
Fork the lambda-dynamodb-local project, or find me on GitHub and let me know how you’d like to help out.
AWS Lambda and Amazon API Gateway launch in Frankfurt region
Vyom Nagrani, Sr. Product Manager, AWS Lambda
We’re happy to announce that you can now build and deploy serverless applications using AWS Lambda and Amazon API Gateway in the Frankfurt region.
Amazon S3, Amazon Kinesis, Amazon SNS, Amazon DynamoDB Streams, Amazon CloudWatch Events, Amazon CloudWatch Logs, and Amazon API Gateway are available as event sources in the Frankfurt region. You can now trigger a Lambda function to process your data stored in Germany using any of these AWS services.
Using Amazon API Gateway as a proxy for DynamoDB
Andrew Baird, AWS Solutions Architect
Amazon API Gateway has a feature that enables customers to create their own API definitions directly in front of an AWS service API. This tutorial will walk you through an example of doing so with Amazon DynamoDB.
Why use API Gateway as a proxy for AWS APIs?
Many AWS services provide APIs that applications depend on directly for their functionality. Examples include:
- Amazon DynamoDB – An API-accessible NoSQL database.
- Amazon Kinesis – Real-time ingestion of streaming data via API.
- Amazon CloudWatch – API-driven metrics collection and retrieval.
If AWS already exposes internet-accessible APIs, why would you want to use API Gateway as a proxy for them? Why not allow applications to just directly depend on the AWS service API itself?
Here are a few great reasons to do so:
- You might want to enable your application to integrate with very specific functionality that an AWS service provides, without the need to manage access keys and secret keys that AWS APIs require.
- There may be application-specific restrictions you’d like to place on the API calls being made to AWS services that you would not be able to enforce if clients integrated with the AWS APIs directly.
- You may get additional value out of using a different HTTP method from the method that is used by the AWS service. For example, creating a GET request as a proxy in front of an AWS API that requires an HTTP POST so that the response will be cached.
- You can accomplish the above things without having to introduce a server-side application component that you need to manage or that could introduce increased latency. Even a lightweight Lambda function that calls a single AWS service API is code that you do not need to create or maintain if you use API Gateway directly as an AWS service proxy.
Here, we will walk you through a hypothetical scenario that shows how to create an Amazon API Gateway AWS service proxy in front of Amazon DynamoDB.
The Scenario
You would like the ability to add a public Comments section to each page of your website. To achieve this, you’ll need to accept and store comments and you will need to retrieve all of the comments posted for a particular page so that the UI can display them.
We will show you how to implement this functionality by creating a single table in DynamoDB, and creating the two necessary APIs using the AWS service proxy feature of Amazon API Gateway.
Defining the APIs
The first step is to map out the APIs that you want to create. For both APIs, we’ve linked to the DynamoDB API documentation. Take note of how the API you define below differs in request/response details from the native DynamoDB APIs.
Post Comments
First, you need an API that accepts user comments and stores them in the DynamoDB table. Here’s the API definition you’ll use to implement this functionality:
Resource: /comments
HTTP Method: POST
HTTP Request Body:
{
"pageId": "example-page-id",
"userName": "ExampleUserName",
"message": "This is an example comment to be added."
}
After you create it, this API becomes a proxy in front of the DynamoDB API PutItem.
Get Comments
Second, you need an API to retrieve all of the comments for a particular page. Use the following API definition:
Resource: /comments/{pageId}
HTTP Method: GET
The curly braces around {pageId} in the URI path definition indicate that pageId will be treated as a path variable within the URI.
This API will be a proxy in front of the DynamoDB API Query. Here, you will notice the benefit: your API uses the GET method, while the DynamoDB GetItem API requires an HTTP POST and does not include any cache headers in the response.
Creating the DynamoDB Table
First, Navigate to the DynamoDB console and select Create Table. Next, name the table Comments, with commentId as the Primary Key. Leave the rest of the default settings for this example, and choose Create.
After this table is populated with comments, you will want to retrieve them based on the page that they’ve been posted to. To do this, create a secondary index on an attribute called pageId. This secondary index enables you to query the table later for all comments posted to a particular page. When viewing your table, choose the Indexes tab and choose Create index.
When querying this table, you only want to retrieve the pieces of information that matter to the client: in this case, these are the pageId, the userName, and the message itself. Any other data you decide to store with each comment does not need to be retrieved from the table for the publically accessible API. Type the following information into the form to capture this and choose Create index:
Creating the APIs
Now, using the AWS service proxy feature of Amazon API Gateway, we’ll demonstrate how to create each of the APIs you defined. Navigate to the API Gateway service console, and choose Create API. In API name, type CommentsApi and type a short description. Finally, choose Create API.
Now you’re ready to create the specific resources and methods for the new API.
Creating the Post Comments API
In the editor screen, choose Create Resource. To match the description of the Post Comments API above, provide the appropriate details and create the first API resource:
Now, with the resource created, set up what happens when the resource is called with the HTTP POST method. Choose Create Method and select POST from the drop down. Click the checkmark to save.
To map this API to the DynamoDB API needed, next to Integration type, choose Show Advanced and choose AWS Service Proxy.
Here, you’re presented with options that define which specific AWS service API will be executed when this API is called, and in which region. Fill out the information as shown, matching the DynamoDB table you created a moment ago. Before you proceed, create an AWS Identity and Access Management (IAM) role that has permission to call the DynamoDB API PutItem for the Comments table; this role must have a service trust relationship to API Gateway. For more information on IAM policies and roles, see the Overview of IAM Policies topic.
After inputting all of the information as shown, choose Save.
If you were to deploy this API right now, you would have a working service proxy API that only wraps the DynamoDB PutItem API. But, for the Post Comments API, you’d like the client to be able to use a more contextual JSON object structure. Also, you’d like to be sure that the DynamoDB API PutItem is called precisely the way you expect it to be called. This eliminates client-driven error responses and removes the possibility that the new API could be used to call another DynamoDB API or table that you do not intend to allow.
You accomplish this by creating a mapping template. This enables you to define the request structure that your API clients will use, and then transform those requests into the structure that the DynamoDB API PutItem requires.
From the Method Execution screen, choose Integration Request:
In the Integration Request screen expand the Mapping Templates section and choose Add mapping template. Under Content-Type, type application/json and then choose the check mark:
Next, choose the pencil icon next to Input passthrough and choose Mapping template from the dropdown. Now, you’ll be presented with a text box where you create the mapping template. For more information on creating mapping templates, see API Gateway Mapping Template Reference.
The mapping template will be as follows. We’ll walk through what’s important about it next:
{
"TableName": "Comments",
"Item": {
"commentId": {
"S": "$context.requestId"
},
"pageId": {
"S": "$input.path('$.pageId')"
},
"userName": {
"S": "$input.path('$.userName')"
},
"message": {
"S": "$input.path('$.message')"
}
}
}
This mapping template creates the JSON structure required by the DynamoDB PutItem API. The entire mapping template is static. The three input variables are referenced from the request JSON using the $input variable and each comment is stamped with a unique identifier. This unique identifier is the commentId and is extracted directly from the API request’s $context variable. This $context variable is set by the API Gateway service itself. To review other parameters that are available to a mapping template, see API Gateway Mapping Template Reference. You may decide that including information like sourceIp or other headers could be valuable to you.
With this mapping template, no matter how your API is called, the only variance from the DynamoDB PutItem API call will be the values of pageId, userName, and message. Clients of your API will not be able to dictate which DynamoDB table is being targeted (because “Comments” is statically listed), and they will not have any control over the object structure that is specified for each item (each input variable is explicitly declared a string to the PutItem API).
Back in the Method Execution pane click TEST.
Create an example Request Body that matches the API definition documented above and then choose Test. For example, your request body could be:
{
"pageId": "breaking-news-story-01-18-2016",
"userName": "Just Saying Thank You",
"message": "I really enjoyed this story!!"
}
Navigate to the DynamoDB console and view the Comments table to show that the request really was successfully processed:
Great! Try including a few more sample items in the table to further test the Get Comments API.
If you deployed this API, you would be all set with a public API that has the ability to post public comments and store them in DynamoDB. For some use cases you may only want to collect data through a single API like this: for example, when collecting customer and visitor feedback, or for a public voting or polling system. But for this use case, we’ll demonstrate how to create another API to retrieve records from a DynamoDB table as well. Many of the details are similar to the process above.
Creating the Get Comments API
Return to the Resources view, choose the /comments resource you created earlier and choose Create Resource, like before.
This time, include a request path parameter to represent the pageId of the comments being retrieved. Input the following information and then choose Create Resource:
In Resources, choose your new /{pageId} resource and choose Create Method. The Get Comments API will be retrieving data from our DynamoDB table, so choose GET for the HTTP method.
In the method configuration screen choose Show advanced and then select AWS Service Proxy. Fill out the form to match the following. Make sure to use the appropriate AWS Region and IAM execution role; these should match what you previously created. Finally, choose Save.
Modify the Integration Request and create a new mapping template. This will transform the simple pageId path parameter on the GET request to the needed DynamoDB Query API, which requires an HTTP POST. Here is the mapping template:
{
"TableName": "Comments",
"IndexName": "pageId-index",
"KeyConditionExpression": "pageId = :v1",
"ExpressionAttributeValues": {
":v1": {
"S": "$input.params('pageId')"
}
}
}
Now test your mapping template. Navigate to the Method Execution pane and choose the Test icon on the left. Provide one of the pageId values that you’ve inserted into your Comments table and choose Test.
You should see a response like the following; it is directly passing through the raw DynamoDB response:
Now you’re close! All you need to do before you deploy your API is to map the raw DynamoDB response to the similar JSON object structure that you defined on the Post Comment API.
This will work very similarly to the mapping template changes you already made. But you’ll configure this change on the Integration Response page of the console by editing the default mapping response’s mapping template.
Navigate to Integration Response and expand the 200 response code by choosing the arrow on the left. In the 200 response, expand the Mapping Templates section. In Content-Type choose application/json then choose the pencil icon next to Output Passthrough.
Now, create a mapping template that extracts the relevant pieces of the DynamoDB response and places them into a response structure that matches our use case:
#set($inputRoot = $input.path('$'))
{
"comments": [
#foreach($elem in $inputRoot.Items) {
"commentId": "$elem.commentId.S",
"userName": "$elem.userName.S",
"message": "$elem.message.S"
}#if($foreach.hasNext),#end
#end
]
}
Now choose the check mark to save the mapping template, and choose Save to save this default integration response. Return to the Method Execution page and test your API again. You should now see a formatted response.
Now you have two working APIs that are ready to deploy! See our documentation to learn about how to deploy API stages.
But, before you deploy your API, here are some additional things to consider:
- Authentication: you may want to require that users authenticate before they can leave comments. Amazon API Gateway can enforce IAM authentication for the APIs you create. To learn more, see Amazon API Gateway Access Permissions.
- DynamoDB capacity: you may want to provision an appropriate amount of capacity to your Comments table so that your costs and performance reflect your needs.
- Commenting features: Depending on how robust you’d like commenting to be on your site, you might like to introduce changes to the APIs described here. Examples are attributes that track replies or timestamp attributes.
Conclusion
Now you’ve got a fully functioning public API to post and retrieve public comments for your website. This API communicates directly with the Amazon DynamoDB API without you having to manage a single application component yourself!
Using Amazon API Gateway with microservices deployed on Amazon ECS
Rudy Krol, AWS Solutions Architect
One convenient way to run microservices is to deploy them as Docker containers. Docker containers are quick to provision, easily portable, and provide process isolation. Amazon EC2 Container Service (Amazon ECS) provides a highly scalable, high performance container management service. This service supports Docker containers and enables you to easily run microservices on a managed cluster of Amazon EC2 instances.
Microservices usually expose REST APIs for use in front ends, third-party applications, and other microservices. A best practice is to manage these APIs with an API gateway. This provides a unique entry point for all of your APIs and also eliminates the need to implement API-specific code for things like security, caching, throttling, and monitoring for each of your microservices. You can implement this pattern in a few minutes using Amazon API Gateway. Amazon API Gateway is a fully managed service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale.
In this post, we’ll explain how to use Amazon API Gateway to expose APIs for microservices running on Amazon ECS by leveraging the HTTP proxy mode of Amazon API Gateway. Amazon API Gateway can make proxy calls to any publicly accessible endpoint; for example, an Elastic Load Balancing load balancer endpoint in front of a microservice that is deployed on Amazon ECS. The following diagram shows the high level architecture described in this article:
You will see how you can benefit from stage variables to dynamically set the endpoint value depending on the stage of the API deployment.
In the first part of this post, we will walk through the AWS Management Console to create the dev environment (ECS cluster, ELB load balancers, and API Gateway configuration). The second part explains how to automate the creation of a production environment with AWS CloudFormation and AWS CLI.
Creating a dev environment with the AWS Management Console
Let’s begin by provisioning a sample helloworld microservice using the Getting Started wizard.
Sign in to Amazon ECS console. If this is the first time you’re using the Amazon ECS console, you’ll see a welcome page. Otherwise, you’ll see the console home page and the Create Cluster button.
Step 1: Create a task definition
- In the Amazon ECS console, do one of the following:
- If Get Started Now is displayed, choose it.
- If it is not displayed, go to the Getting Started wizard.
- Optional: (depending on the AWS Region) Deselect the Store container images securely with Amazon ECR checkbox and choose Continue.
- For Task definition name, type
ecsconsole-helloworld. - For Container name, type
helloworld. - Choose Advanced options and type the following text in the Command field:
/bin/sh -c "echo '{ \"hello\" : \"world\" }' > /usr/local/apache2/htdocs/index.html && httpd-foreground" - Choose Update and then choose Next step
Step 2: Configure service
- For Service name, type
ecsconsole-service-helloworld. - For Desired number of tasks, type
2. - In the Elastic load balancing section, for Container name: host port, choose
helloworld:80. - For Select IAM role for service, choose Create new role or use an existing
ecsServiceRoleif you already created the required role. - Choose Next Step.
Step 3: Configure cluster
- For Cluster name, type
dev. - For Number of instances, type
2. - For Select IAM role for service, choose Create new role or use an existing
ecsInstanceRoleif you already created the required role. - Choose Review and Launch and then choose Launch Instance & Run Service.
At this stage, after a few minutes of pending process, the helloworld microservice will be running in the dev ECS cluster with an ELB load balancer in front of it. Make note of the DNS Name of the ELB load balancer for later use; you can find it in the Load Balancers section of the EC2 console.
Configuring API Gateway
Now, let’s configure API Gateway to expose the APIs of this microservice. Sign in to the API Gateway console. If this is your first time using the API Gateway console, you’ll see a welcome page. Otherwise, you’ll see the API Gateway console home page and the Create API button.
Step 1: Create an API
- In the API Gateway console, do one of the following:
- If Get Started Now is displayed, choose it.
- If Create API is displayed, choose it.
- If neither is displayed, in the secondary navigation bar, choose the API Gateway console home button, and then choose Create API.
- For API name, type
EcsDemoAPI. - Choose Create API.
Step 2: Create Resources
- In the API Gateway console, choose the root resource (/), and then choose Create Resource.
- For Resource Name, type
HelloWorld. - For Resource Path, leave the default value of
/helloworld. - Choose Create Resource.
Step 3: Create GET Methods
- In the Resources pane, choose /helloworld, and then choose Create Method.
- For the HTTP method, choose GET, and then save your choice.
Step 4: Specify Method Settings
- In the Resources pane, in /helloworld, choose GET.
- In the Setup pane, for Integration type, choose HTTP Proxy.
- For HTTP method, choose GET.
- For Endpoint URL, type
http://${stageVariables.helloworldElb} - Choose Save.
Step 5: Deploy the API
- In the Resources pane, choose Deploy API.
- For Deployment stage, choose New Stage.
- For Stage name, type
dev. - Choose Deploy.
- In the stage settings page, choose the Stage Variables tab.
- Choose Add Stage Variable, type
helloworldElbfor Name, type the DNS Name of the ELB in the Value field and then save.
Step 6: Test the API
- In the Stage Editor pane, next to Invoke URL, copy the URL to the clipboard. It should look something like this:
https://.execute-api..amazonaws.com/dev - Paste this URL in the address box of a new browser tab.
- Append
/helloworldto the URL and validate. You should see the following JSON document:{ "hello": "world" }
Automating prod environment creation
Now we’ll improve this setup by automating the creation of the prod environment. We use AWS CloudFormation to set up the prod ECS cluster, deploy the helloworld service, and create an ELB in front of the service. You can use the template with your preferred method:
Using AWS CLI
aws cloudformation create-stack --stack-name EcsHelloworldProd --template-url https://s3.amazonaws.com/rko-public-bucket/ecs_cluster.template --parameters ParameterKey=AsgMaxSize,ParameterValue=2 ParameterKey=CreateElasticLoadBalancer,ParameterValue=true ParameterKey=EcsInstanceType,ParameterValue=t2.micro
Using AWS console
Launch the AWS CloudFormation stack with the Launch Stack button below and use these parameter values:
- AsgMaxSize:
2 - CreateElasticLoadBalancer:
true - EcsInstanceType:
t2.micro
Configuring API Gateway with AWS CLI
We’ll use the API Gateway configuration that we created earlier and simply add the prod stage.
Here are the commands to create the prod stage and configure the stage variable to point to the ELB load balancer:
#Retrieve API ID
API_ID=$(aws apigateway get-rest-apis --output text --query "items[?name=='EcsDemoAPI'].{ID:id}")
#Retrieve ELB DNS name from CloudFormation Stack outputs
ELB_DNS=$(aws cloudformation describe-stacks --stack-name EcsHelloworldProd --output text --query "Stacks[0].Outputs[?OutputKey=='EcsElbDnsName'].{DNS:OutputValue}")
#Create prod stage and set helloworldElb variable
aws apigateway create-deployment --rest-api-id $API_ID --stage-name prod --variables helloworldElb=$ELB_DNS
You can then test the API on the prod stage using this simple cURL command:
AWS_REGION=$(aws configure get region) curl https://$API_ID.execute-api.$AWS_REGION.amazonaws.com/prod/helloworld
You should see { "hello" : "world" } as the result of the cURL request. If the result is an error message like {"message": "Internal server error"}, verify that you have healthy instances behind your ELB load balancer. It can take some time to pass the health checks, so you’ll have to wait for a minute before trying again.
From the stage settings page you also have the option to export the API configuration to a Swagger file, including the API Gateway extension. Exporting the API configuration as a Swagger file enables you to keep the definition in your source repository. You can then import it at any time, either by overwriting the existing API or by importing it as a brand new API. The API Gateway import tool helps you parse the Swagger definition and import it into the service.
Conclusion
In this post, we looked at how to use Amazon API Gateway to expose APIs for microservices deployed on Amazon ECS. The integration with the HTTP proxy mode pointing to ELB load balancers is a simple method to ensure the availability and scalability of your microservice architecture. With ELB load balancers, you don’t have to worry about how your containers are deployed on the cluster.
We also saw how stage variables help you connect your APIs on different ELB load balancers, depending on the stage where the API is deployed.
Scheduling SSH jobs using AWS Lambda
Puneet Agarwal, AWS Solution Architect
With the addition of the Scheduled Events feature, you can now set up AWS Lambda to invoke your code on a regular, scheduled basis. You can now schedule various AWS API activities in your account (such as creation or deletion of CloudFormation stacks, EBS volume snapshots, etc.) with AWS Lambda. In addition, you can use AWS Lambda to connect to your Linux instances by using SSH and run desired commands and scripts at regular time intervals. This is especially useful for scheduling tasks (e.g., system updates, log cleanups, maintenance tasks) on your EC2 instances, when you don’t want to manage cron or external schedulers for a dynamic fleet of instances.
In the following example, you will run a simple shell script that prints “Hello World” to an output file on instances tagged as “Environment=Dev” in your account. You will trigger this shell script through a Lambda function written in Python 2.7.
At a high level, this is what you will do in this example:
- Create a Lambda function to fetch IP addresses of EC2 instances with “Environment=Dev” tag. This function will serve as a trigger function. This trigger function will invoke a worker function, for each IP address. The worker function will connect to EC2 instances using SSH and run a HelloWorld.sh script.
- Configure Scheduled Event as an event source to invoke the trigger function every 15 minutes.
- Create a Python deployment package (.zip file), with worker function code and other dependencies.
- Upload the worker function package to AWS Lambda.
Advantages of Scheduled Lambda Events over Ubiquitous Cron
Cron is indeed simple and well understood, which makes it a very popular tool for running scheduled operations. However, there are many architectural benefits that make scheduled Lambda functions and custom scripts a better choice in certain scenarios:
- Decouple job schedule and AMI: If your cron jobs are part of an AMI, each schedule change requires you to create a new AMI version, and update existing instances running with that AMI. This is both cumbersome and time-consuming. Using scheduled Lambda functions, you can keep the job schedule outside of your AMI and change the schedule on the fly.
- Flexible targeting of EC2 instances: By abstracting the job schedule from AMI and EC2 instances, you can flexibly target a subset of your EC2 instance fleet based on tags or other conditions. In this example, we are targeting EC2 instances with the “Environment=Dev” tag.
- Intelligent scheduling: With scheduled Lambda functions, you can add custom logic to you abstracted job scheduler.
While there are many ways of achieving the above benefits, scheduled Lambda functions are an easy-to-use option in your toolkit.
Trigger Function
This is a simple Python function that extracts IP addresses of all instances with the “Environment=Dev” tag and invokes the worker function for each of the instances. Decoupling the trigger function from the worker function enables a simpler programming model for parallel execution of tasks on multiple instances.
Steps:
- Sign in to the AWS Management Console and open the AWS Lambda console.
- Choose Create a Lambda function.
- On the Select blueprint page, type cron in the search box.
- Choose lambda-canary.
- On the Configure event sources page, Event source type defaults to Scheduled Event. You can create a new schedule by entering a name for the schedule, or can select one of your existing schedules. For Schedule expression, you can specify a fixed rate (number of minutes, hours, or days between invocations) or you can specify a cron-like expression. Note that rate frequencies of less than five minutes are not supported at this time.
- Choose Next. The Configure Function page appears.
Here, you can enter the name and description of your function. Replace the sample code here with the following code.
trigger_function.pyimport boto3 def trigger_handler(event, context): #Get IP addresses of EC2 instances client = boto3.client('ec2') instDict=client.describe_instances( Filters=[{'Name':'tag:Environment','Values':['Dev']}] ) hostList=[] for r in instDict['Reservations']: for inst in r['Instances']: hostList.append(inst['PublicIpAddress']) #Invoke worker function for each IP address client = boto3.client('lambda') for host in hostList: print "Invoking worker_function on " + host invokeResponse=client.invoke( FunctionName='worker_function', InvocationType='Event', LogType='Tail', Payload='{"IP":"'+ host +'"}' ) print invokeResponse return{ 'message' : "Trigger function finished" } - After adding the trigger code in the console, create the appropriate execution role and set a timeout. Note that the execution role must have permissions to execute EC2 DescribeInstances and invoke Lambda functions. Example IAM Policies for the trigger Lambda role are as follows:
- Basic execution policy: https://gist.github.com/apun/8f8c0c0cbea38d7e0bdc (automatically created by AWS Console).
- Trigger Policy: https://gist.github.com/apun/33c2fd954a8e238bbcb0 (EC2:Describe* and InvokeFunction permissions to invoke worker_function). After role creation, you can add this policy to the trigger_lambda_role using the IAM console.
- Choose Next, choose Enable later, and then choose Create function.
Worker Function
Next, put together the worker Lambda function that connects to an Amazon EC2 instance using SSH, and then run the HelloWorld.sh script. To initiate SSH connections from the Lambda client, use the Paramiko library. Paramiko is an open source Python implementation of the SSHv2 protocol, providing both client and server functionality. Worker function will irst download a private key file from a secured Amazon S3 bucket to the local /tmp folder, and then use that key file to connect to the EC2 instances by using SSH. You must keep your private key secure and make sure that only the worker function has read access to the file on S3. Assuming that EC2 instances have S3 access permissions through an EC2 role, worker function will download the HelloWorld.sh script from S3 and execute it locally on each EC2 instance.
Steps:
- Create worker_function.py file on your local Linux machine or on an EC2 instance using following code
worker_function.pyimport boto3 import paramiko def worker_handler(event, context): s3_client = boto3.client('s3') #Download private key file from secure S3 bucket s3_client.download_file('s3-key-bucket','keys/keyname.pem', '/tmp/keyname.pem') k = paramiko.RSAKey.from_private_key_file("/tmp/keyname.pem") c = paramiko.SSHClient() c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) host=event['IP'] print "Connecting to " + host c.connect( hostname = host, username = "ec2-user", pkey = k ) print "Connected to " + host commands = [ "aws s3 cp s3://s3-bucket/scripts/HelloWorld.sh /home/ec2-user/HelloWorld.sh", "chmod 700 /home/ec2-user/HelloWorld.sh", "/home/ec2-user/HelloWorld.sh" ] for command in commands: print "Executing {}".format(command) stdin , stdout, stderr = c.exec_command(command) print stdout.read() print stderr.read() return { 'message' : "Script execution completed. See Cloudwatch logs for complete output" }
Now, creating a deployment package is straightforward. For this example, create a deployment package using Virtualenv. - Install Virtualenv on your local Linux machine or an EC2 instance.
$ pip install virtualenv
- Create a virtual environment named “helloworld-env“, which will use a Python2.7 interpreter.
$ virtualenv –p /usr/bin/python2.7 path/to/my/helloworld-env
- Activate helloworld-env.
source path/to/my/helloworld-env/bin/activate
- Install dependencies.
$pip install pycrypto
PyCrypto provides the low-level (C-based) encryption algorithms we need to implement the SSH protocol.
$pip install paramiko
- Add worker_function.py to the zip file.
$zip path/to/zip/worker_function.zip worker_function.py
- Add dependencies from helloworld-env to the zip file.
$cd path/to/my/helloworld-env/lib/python2.7/site-packages $zip –r path/to/zip/worker_function.zip $cd path/to/my/helloworld-env/lib64/python2.7/site-packages $zip –r path/to/zip/worker_function.zip
Using the AWS console (skip the blueprint step) or AWS CLI, create a new Lambda function named worker_function and upload worker_function.zip.
Example IAM policies for the worker Lambda role are as follows:- Basic execution policy: https://gist.github.com/apun/8f8c0c0cbea38d7e0bdc (Automatically created by AWS Console)
- Worker policy: https://gist.github.com/apun/0647280645b399917191 (GetObject permission for S3 key file)
Caution: To keep your keys secure, make sure no other IAM users or roles, other than intended users, have access to this S3 bucket.
Upload key and script to S3
All you need to do now is upload your key and script file to S3 buckets and then you are ready to run the example.
Steps:
- Upload HellowWorld.sh to an appropriate S3 bucket (e.g., s3://s3-bucket/scripts/). HelloWorld.sh is a simple shell script that prints “Hello World from instanceID” to a log file and copies that log file to your S3 folder.
HelloWorld.sh#Get instanceId from metadata instanceid=`wget -q -O - http://instance-data/latest/meta-data/instance-id` LOGFILE="/home/ec2-user/$instanceid.$(date +"%Y%m%d_%H%M%S").log" #Run Hello World and redirect output to a log file echo "Hello World from $instanceid" > $LOGFILE #Copy log file to S3 logs folder aws s3 cp $LOGFILE s3://s3-bucket/logs/
- Upload keyname.pem file, which is your private key to connect to EC2 instances, to a secure S3 bucket (e.g., s3://s3-key-bucket/keys/keyname.pem). To keep your keys secure, make sure no IAM users or roles, other than intended users and the Lambda worker role, have access to this S3 bucket.
Running the example
As a final step, enable your trigger_function event source by choosing trigger_function from the list of Lambda functions, choosing the Event sources tab, and clicking Disabled in the State column.
You can now test your newly created Lambda functions and monitor execution logs. AWS Lambda logs all requests handled by your function and automatically stores logs generated by your code using Amazon CloudWatch Logs. The following screenshots show my CloudWatch Logs after completing the preceding steps.
Trigger function log in CloudWatch Logs:
Worker function log in Cloudwatch Logs:
Log files that were generated in my S3 bucket:
Other considerations
- With the new Lambda VPC support, you can connect to your EC2 instances running in your private VPC by providing private subnet IDs and EC2 security group IDs as part of your Lambda function configuration.
- AWS Lambda now supports a maximum function duration of 5 minutes, and so you can use scheduled Lambda functions to run jobs that are expected to finish within 5 minutes. For longer running jobs, you can use following syntax to run jobs in the background so that the Lambda function doesn’t wait for command execution to finish.
c.exec_command(cmd + ' > /dev/null 2>&1 &')
Introducing custom authorizers in Amazon API Gateway
Today Amazon API Gateway is launching custom request authorizers. With custom request authorizers, developers can authorize their APIs using bearer token authorization strategies, such as OAuth using an AWS Lambda function. For each incoming request, API Gateway verifies whether a custom authorizer is configured, and if so, API Gateway calls the Lambda function with the authorization token. You can use Lambda to implement various authorization strategies (e.g., JWT verification, OAuth provider callout). Custom authorizers must return AWS Identity and Access Management (IAM) policies. These policies are used to authorize the request. If the policy returned by the authorizer is valid, API Gateway caches the returned policy associated with the incoming token for up to 1 hour so that your Lambda function doesn’t need to be invoked again.
Configuring custom authorizers
You can configure custom authorizers from the API Gateway console or using the APIs. In the console, we have added a new section called custom authorizers inside your API.
An API can have multiple custom authorizers and each method within your API can use a different authorizer. For example, the POST method for the /login resource can use a different authorizer than the GET method for the /pets resource.
To configure an authorizer you must specify a unique name and select a Lambda function to act as the authorizer. You also need to indicate which field of the incoming request contains your bearer token. API Gateway will pass the value of the field to your Lambda authorizer. For example, in most cases your bearer token will be in the Authorization header; you can select this field using the method.request.header.Authorization mapping expression. Optionally, you can specify a regular expression to validate the incoming token before your authorizer is triggered and you can also specify a TTL for the policy cache.
Once you have configured a custom authorizer, you can simply select it from the authorization dropdown in the method request page.
The authorizer function in AWS Lambda
API Gateway invokes the Lambda authorizer by passing in the Lambda event. The Lambda event includes the bearer token from the request and full ARN of the API method being invoked. The authorizer Lambda event looks like this:
{
"type":"TOKEN",
"authorizationToken":"<Incoming bearer token>",
"methodArn":"arn:aws:execute-api:<Region id>:<Account id>:<API id>/<Stage>/<Method>/<Resource path>"
}
Your Lambda function must return a valid IAM policy. API Gateway uses this policy to make authorization decisions for the token. For example, if you use JWT tokens, you can use the Lambda function to open the token and then generate a policy based on the scopes included in the token. Later today we will publish authorizer Lambda blueprints for Node.js and Python that include a policy generator object. This sample function uses AWS Key Management Service (AWS KMS) to decrypt the signing key for the token, the nJwt library for Node.js to validate a token, and then the policy generator object included in the Lambda blueprint to generate and return a valid policy to Amazon API Gateway.
var nJwt = require('njwt');
var AWS = require('aws-sdk');
var signingKey = "CiCnRmG+t+ BASE 64 ENCODED ENCRYPTED SIGNING KEY Mk=";
exports.handler = function(event, context) {
console.log('Client token: ' + event.authorizationToken);
console.log('Method ARN: ' + event.methodArn);
var kms = new AWS.KMS();
var decryptionParams = {
CiphertextBlob : new Buffer(signingKey, 'base64')
}
kms.decrypt(decryptionParams, function(err, data) {
if (err) {
console.log(err, err.stack);
context.fail("Unable to load encryption key");
} else {
key = data.Plaintext;
try {
verifiedJwt = nJwt.verify(event.authorizationToken, key);
console.log(verifiedJwt);
// parse the ARN from the incoming event
var apiOptions = {};
var tmp = event.methodArn.split(':');
var apiGatewayArnTmp = tmp[5].split('/');
var awsAccountId = tmp[4];
apiOptions.region = tmp[3];
apiOptions.restApiId = apiGatewayArnTmp[0];
apiOptions.stage = apiGatewayArnTmp[1];
policy = new AuthPolicy(verifiedJwt.body.sub, awsAccountId, apiOptions);
if (verifiedJwt.body.scope.indexOf("admins") > -1) {
policy.allowAllMethods();
} else {
policy.allowMethod(AuthPolicy.HttpVerb.GET, "*");
policy.allowMethod(AuthPolicy.HttpVerb.POST, "/users/" + verifiedJwt.body.sub);
}
context.succeed(policy.build());
} catch (ex) {
console.log(ex, ex.stack);
context.fail("Unauthorized");
}
}
});
};
You can also generate a policy in your code instead of using the provided AuthPolicy object. Valid policies include the principal identifier associated with the token and a named IAM policy that can be cached and used to authorize future API calls with the same token. The principalId will be accessible in the mapping template.
{
"principalId": "xxxxxxx", // the principal user identification associated with the token send by the client
"policyDocument": { // example policy shown below, but this value is any valid policy
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"execute-api:Invoke"
],
"Resource": [
"arn:aws:execute-api:us-east-1:xxxxxxxxxxxx:xxxxxxxx:/test/*/mydemoresource/*"
]
}
]
}
}
To learn more about the possible options in a policy, see the public access permissions reference for API Gateway. All of the variables that are normally available in IAM policies are also available to custom authorizer policies. For example, you could restrict access using the ${aws:sourceIp} variable. To learn more, see the policy variables reference.
Because policies are cached for a configured TTL, API Gateway only invokes your Lambda function the first time it sees a token; all of the calls that follow during the TTL period are authorized by API Gateway using the cached policy.
Conclusion
You can use custom authorizers in API Gateway to support any bearer token. This allows you to authorize access to your APIs using tokens from an OAuth flow or SAML assertions. Further, you can leverage all of the variables available to IAM policies without setting up your API to use IAM authorization.
Custom authorizers are available in the API Gateway console and APIs now, and authorizer Lambda blueprints will follow later today. Get in touch through the API Gateway forum if you have questions or feedback about custom authorizers.
