AWS Blog

My colleague Eric Fitzgerald wrote the guest post below in order to show you how to use an AWS Lambda function to forward Amazon Inspector findings to your ticketing and workflow systems.

At AWS Re:Invent 2015 we announced Amazon Inspector, our security vulnerability assessment service that helps customers test for security vulnerabilities early and often.  Using Amazon Inspector, customers can automate security testing across development, test, and production environments, identifying security vulnerabilities as part of the entire software development, deployment, and operations lifecycle.

Customer feedback on the Amazon Inspector approach to automated security testing has been overwhelming positive.  Customers have told us that with Amazon Inspector, they are able to run security assessments more frequently and are catching security vulnerabilities earlier than they have in the past.  However, identifying the security vulnerabilities is only half the battle, the vulnerabilities that are found need to be remediated. Many of our customers have started to integrate Amazon Inspector with their workflow and ticketing systems in order to automate and accelerate the remediation workflow for Amazon Inspector findings.  We designed Amazon Inspector with this in mind and thought we would share more detail on one method for integrating Amazon Inspector findings with email, workflow, and ticketing systems.

Using AWS Lambda to push Amazon Inspector Findings to a Ticketing System
In this example, we are using an AWS Lambda function to connect Amazon Inspector to systems that can handle incident creation via email. Here’s the chain of events:

1. Amazon Inspector runs and performs a security assessment. It sends a message to an Amazon Simple Notification Service (SNS) topic at the end of the run.
2. The Lambda function is invoked by the SNS message.
3. The function fetches the findings from the security assessment.
4. The function formats and emails the findings using another SNS topic.

Along the way, the function creates the destination topic and the email subscription if necessary.

Setting up the Function
You will need to set up the function in the AWS Region where you run your Amazon Inspector assessments. If you run Amazon Inspector in more than one region, you’ll need to repeat the steps for each one. Here are the steps:

1. Create the SNS topic for Amazon Inspector.
2. Configure Amazon Inspector to send findings to the newly created topic.
3. Set up the Lambda function to fetch, format, and email the findings.

Configure an SNS Topic
The first major step is to configure an Amazon SNS topic that Amazon Inspector will notify when there are new findings, and an Amazon SNS topic that will format and send findings as email to other systems.

Navigate to the Amazon SNS Console and create a new Amazon SNS topic.  This will be the topic where Amazon Inspector will deliver notifications to.  It does not matter what you name the topic.

Next, assign the following policy to the topic.  You can do this in the Amazon SNS Console by selecting the topic, clicking on Other topic actions, and selecting Edit topic policy.  In the advanced view, replace the existing policy text with this policy:

{
  "Version": "2008-10-17",
  "Id": "inspector-sns-publish-policy",
  "Statement": [
    {
      "Sid": "inspector-sns-publish-statement",
      "Effect": "Allow",
      "Principal": {
        "Service": "inspector.amazonaws.com"
      },
      "Action": "SNS:Publish",
      "Resource": "arn:aws:sns:*"
    }
  ]
}

If you are familiar with AWS Identity and Access Management (IAM) policies, then a security best practice is to change the value of the Resource field of the policy to exactly match the Amazon SNS topic ARN, in order to restrict Amazon Inspector so that it can only publish to this topic.

Configure Amazon Inspector
Navigate to the Amazon Inspector Console, visit the Assessment templates page, and select the assessment template whose findings you want sent to the external system.  Expand the row, and you’ll see a section called SNS topics.  Click the pencil icon to the left of the Amazon SNS topics section and you’ll be able to pick the Amazon SNS topic you just created from a drop-down list.  Once you’ve selected the topic, click on Save.

Set up the Lambda Function
Navigate to the Lambda Console and create a new function using the SNS-message-python blueprint:

Select SNS for the event source and then select the SNS topic that you created in the first step:

To finish configuring the function, click Next.  Type a name and description for the function, choose the Python 2.7 runtime, and replace the sample function code with this code:

from __future__ import print_function
import boto3
import json
import datetime

sns = boto3.client('sns')
inspector = boto3.client('inspector')

# SNS topic - will be created if it does not already exist
SNS_TOPIC = "Inspector-Finding-Delivery"

# Destination email - will be subscribed to the SNS topic if not already
DEST_EMAIL_ADDR = "eric@example.com"

# quick function to handle datetime serialization problems
enco = lambda obj: (
    obj.isoformat()
    if isinstance(obj, datetime.datetime)
    or isinstance(obj, datetime.date)
    else None
)

def lambda_handler(event, context):

    # extract the message that Inspector sent via SNS
    message = event['Records'][0]['Sns']['Message']

    # get inspector notification type
    notificationType = json.loads(message)['event']

    # skip everything except report_finding notifications
    if notificationType != "FINDING_REPORTED":
        print('Skipping notification that is not a new finding: ' + notificationType)
        return 1
    
    # extract finding ARN
    findingArn = json.loads(message)['finding']

    # get finding and extract detail
    response = inspector.describe_findings(findingArns = [ findingArn ], locale='EN_US')
    print(response)
    try:
        finding = response['findings'][0]
    except OSError as err:
        print("OS error: {0}".format(err))
    except:
        print("Unexpected error:", sys.exc_info()[0])
        raise
        
    # skip uninteresting findings
    title = finding['title']
    if title == "Unsupported Operating System or Version":
        print('Skipping finding: ', title)
        return 1
        
    if title == "No potential security issues found":
        print('Skipping finding: ', title)
        return 1
    
    # get the information to send via email
    subject = title[:100] # truncate @ 100 chars, SNS subject limit
    messageBody = "Title:\n" + title + "\n\nDescription:\n" + finding['description'] + "\n\nRecommendation:\n" + finding['recommendation']
    
    # un-comment the following line to dump the entire finding as raw json
    # messageBody = json.dumps(finding, default=enco, indent=2)

    # create SNS topic if necessary
    response = sns.create_topic(Name = SNS_TOPIC)
    snsTopicArn = response['TopicArn']

    # check to see if the subscription already exists
    subscribed = False
    response = sns.list_subscriptions_by_topic( TopicArn = snsTopicArn )

    # iterate through subscriptions array in paginated list API call
    while True:
        for subscription in response['Subscriptions']:
            if ( subscription['Endpoint'] == DEST_EMAIL_ADDR ):
                subscribed = True
                break
        
        if 'NextToken' not in response:
            break
        
        response = sns.list_subscriptions_by_topic(
            TopicArn = snsTopicArn,
            NextToken = response['NextToken']
            )
        
    # create subscription if necessary
    if ( subscribed == False ):
        response = sns.subscribe(
            TopicArn = snsTopicArn,
            Protocol = 'email',
            Endpoint = DEST_EMAIL_ADDR
            )

    # publish notification to topic
    response = sns.publish(
        TopicArn = snsTopicArn,
        Message = messageBody,
        Subject = subject
        )

    return 0

Be sure to edit the DEST_EMAIL_ADDR value, and put in the actual email address that is used to send incidents to your incident management system. Optionally, you can change the name of the SNS topic that Amazon Inspector will use to send findings.

Leave the function handler (lambda_function.lambda_handler) as-is, and give the function a name:

Choose  *basic execution role from the Role drop-down. After Lambda navigates to a new page,  view the policy document, and use this one instead:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:*:*:*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "inspector:DescribeFindings",
                "SNS:CreateTopic",
                "SNS:Subscribe",
                "SNS:ListSubscriptionsByTopic",
                "SNS:Publish"
            ],
            "Resource": "*"
        }
    ]
}

Click the Allow button to create the role and return to AWS Lambda, then leave the advanced settings as-is.

Be sure to click on Enable event source  on the Review page:

Click Create function to save the function.

And that’s it!

Ready to Roll
For any assessments where you want the findings sent to another system, just add the first Amazon SNS topic (the one you created with these instructions) to the assessment template, and ensure that new finding reports are selected for publication to that topic.

The first time you run an assessment, Amazon Inspector will notify Lambda that you have new findings, and the Lambda function that you just created will create the SNS topic (if it doesn’t already exist), subscribe the destination email address to the topic (if not already subscribed), and send the findings as email to that address.  If Lambda had to subscribe the email address to the topic, then you’ll only get one email requiring you to click a link to confirm that you want to subscribe.  After confirmation, Amazon Inspector will deliver findings to that email address.

If you want to connect to Atlassian’s Jira Service Desk, it’s super easy from here on out.  In Jira ServiceDesk, navigate to Customer Channels.  This will display the email address that can receive email and create new issues.  Put that email address into the Lambda function’s Python script and that’s where Inspector will deliver its findings.  ServiceDesk will automatically turn them into ServiceDesk issues, and you can manage your workflow there.

Stay Tuned
Thank you for using Amazon Inspector, and look for more from us soon!

— Eric Fitzgerald, Principal Security Engineer

AWS Marketplace allows AWS customers to find, buy, and immediately start using cloud-based applications developed by Independent Software Vendors (ISVs).  AWS customers collectively rack up 205 million hours per month of AWS Marketplace usage as they make use of over 2,700 offerings from over 925 ISVs.

Support for EU-Based ISVs
ISVs based in the European Union can now register their products in AWS Marketplace without having to create a US-based entity.

The following EU-based ISVs have already listed their products:

BI/Database

• EXASOL AG – EXASOL Analytic Database
• Data Virtuality GmbH – datavirtuality
• Quasardb – quasardb XTP

HPC/Storage

• ThinkParQ GmbH – BeeGFS
• Alces Flight Ltd – Alces Flight Compute
• ABSTRIUM / Pydio – Pydio Enterprise

Security/Monitoring

• Hyperglance – Hyperglance
• MikroTik – Cloud Hosted Router
• PerunWorks – Zenected Threat Defense VPN

Media/Communications

• Cinegy GmbH – Cinegy Air Pro
• Tikal Technologies – elasticRTC

Business Apps

• Fenix Engineering Solutions – nanoAPPs OCS Odoo v8 Spanish Pymes Edition
• Arcusys – Valamis
• Idu Software – idu-Concept

To learn more about their offerings, check our our new Software Solutions from European ISVs page!

Come on In
If you are a US or EU-based ISV and would like to list and sell your products in AWS Marketplace, visit our Sell on AWS Marketplace page.

PS – Other recent feature additions to AWS Marketplace include Support for Clusters and AWS Resources and Additional Pricing Options for Sellers. Also, AWS customers can now request multi-year subscriptions to select products in AWS Marketplace at a negotiated discount from the software vendor (discounts on multi-year subscriptions vary by product and vendor). For more information on the eligible products and vendors, please contact us at aws-mp-bd@amazon.com.

Continuing with our focus on hot AWS-powered startups (March and April), I would like to tell you about three more this month:

• Shaadi.com – Helping South Asians to find a companion for life.
• Capillary – Boosting customer engagement for e-commerce.
• Mondo – A mobile-first bank.

Shaadi.com
Anupam Mittal, founder of Shaadi.com,  was exasperated by the way that marriages were arranged in India. Candidate photos and profiles were spread out on a coffee table and perused in hopes of finding a suitable life partner. He believed that this important, tradition-bound process could be improved, and created Shaadi.com, now one of the world’s largest matchmaking services and one of India’s best-known Internet brands.

Shaadi.com blends time-honored traditions (many going back centuries) with a progressive, consumer-oriented mindset. After having touched the lives of over 35 million people and helping over 4 million people to find their matches, they were recognized as one of the 50 most innovative companies in the world back in 2011.

In order to build a scalable business, Shaadi now runs its production infrastructure on AWS with the assistance of a lean DevOps team. They currently make use of Amazon Elastic Compute Cloud (EC2), Amazon Simple Storage Service (S3), Amazon Simple Notification Service (SNS), Amazon Simple Email Service (SES), and Amazon ElastiCache, with plans to make use of additional managed services in the future. They host their corporate data warehouse on Amazon Redshift and use it to drive all of their reporting and analytics.

Capillary
Back in 2008, the founders of Capillary began to work with retailers to create loyalty programs that were centered around mobile phone numbers. As they did this, they realized that many of the retailers were saddled with traditional, on-premises CRM systems that were not amenable to modernization. The founders stepped in to fill this gap with the goal of creating a cutting-edge customer engagement suite that ran on a multi-tenant cloud-powered platform.

Over the intervening years the solution has grown to encompass CRM, loyalty, e-commerce, customer analytics, and O2O (online-to-offline) commerce. Capillary now connects 150 million shoppers to over 20,000 stores and more than 250 e-commerce implementations across 30 countries, with a focus on driving excellence in online and traditional retail.

Capillary now runs in 5 distinct AWS regions. The architecture is based on microservices and runs on top of EC2, S3, Auto Scaling, Amazon EMR, and ElastiCache, with a focus on security, availability, and scalability (read the Capillary Tech Blog to learn more about how they address these requirements using AWS).

Mondo
Starting with the goal of “building the best bank on the planet,” the team behind UK-based Mondo decided to address the needs of mobile-first users. These users prefer to do their banking via mobile phone instead of in person or on a desktop.  In addition to traditional banking functions, the resulting mobile app can track spending in real time, display geolocated transactions on a map, view spending by category, send money to other users in peer-to-peer fashion, and interact with loyalty programs.  Behind the scenes, the app makes uses of the Mondo API to interact with the actual banking functions.

The founders chose AWS in order to build a scalable and highly reliable system. They practice account separation (distinct accounts for dev, test, staging, and production) and follow the infrastructure-as-code discipline. Because banking is a regulated business, Mondo uses AWS CloudHSM to sign and cryptographically ensure the integrity of payment messages. They use VPCs and Network ACLs to isolate disparate functions and to manage the scope of regulated activities.

My colleague Sanjay Padhi wrote the guest post below in order to tell the story of how AWS powered an important scientific discovery.

Ghostlike particles called neutrinos are everywhere, cosmic rays bombard us with them, the sun bathes us in them. Though incredibly difficult to detect, they may hold the key to why there is more matter than antimatter around. Prominent physicist and Nobel laureate Enrico Fermi named this mysterious particle the “neutrino” (or “little neutral one”). It took time to realize that neutrinos have very unstable egos. They are able to switch or change their identities (“flavors”) through space also called neutrino oscillation. The 2015 Nobel prize in Physics was awarded to Takaaki Kajita (Super-Kamiokanda, Japan) and Arthur B. McDonald (SNO, Canada) for the discovery of neutrino oscillations.

It is extremely exciting to announce that the US flagship experiment in the intensity frontier, NOvA, unveils new results with the help of AWS infrastructure, that sheds light on our understanding of the quantum universe. They see an intriguing preference for “non-maximal” mixing between neutrino identities. These results were presented at the Imperial College in London England, in the presence of researchers from around the world that are gathered for the XXVII International Conference on Neutrino Physics and Astrophysics.

From Experiment to Understanding the Data
Deep in the woods at the Ash River Laboratory in northern Minnesota, close to the Canadian border, the NOvA experiment studies neutrino identities using the most intense particle beams sent from Fermilab, near Chicago, Illinois, passing through the Earth’s crust and traveling over 500 miles. As the neutrinos travel the distance between the laboratories, they undergo a fundamental change in their identities. These changes are carefully measured by the massive NOvA detector.

The detector stands 53 feet tall and 180 feet long and weights 14,000 metric tons (over 30 million pounds). It acts as a gigantic digital camera to observe and capture the faint traces of light and energy that are left by particle interactions within the detector. The experiment captures two million “pictures” per second of these interactions. The pictures are analyzed by sophisticated software. The extreme sensitivity of the detector, electronics and software allow for individual neutrino interactions to be identified, classified and measured.

The NOvA experiment is being conducted by more than 200 scientists from 41 universities and research institutions from seven countries.  It is hosted by Fermilab, the leading U.S. laboratory for high-energy and particle physics.

The new result as shown below is not only consistent with the dramatic change in neutrino identity as seen in previous measurements, but has an intriguing hint that the effect is not quite as large as the theoretically expected maximum. This result is an important part of NOvA’s overall physics goals, and takes a step towards solving the decades long mystery of why there is more matter than antimatter around in the universe we see.

AWS and NOvA
NOvA and other neutrino experiments hosted at Fermilab need to analyze over 10 PB of data each year as part of their ongoing physics analysis efforts. Historically, these analyses have been performed using a combination of dedicated on premises computing centers located at Fermilab, collaborating universities and grid federations. With the increase in data size, complexity of algorithms and the demand for large scale data processing, NOvA via Fermilab’s HEP Cloud used AWS storage and compute infrastructure in order to meet the peak demand for data/IO-intensive processing for the analysis. AWS Cloud Credits for Research Program helped immensely with the adoption and integration towards AWS Cloud.

Amazon S3 for Data Buffering
NOvA ran three major physics analysis campaigns on AWS as part of the critical path to physics results that is presented today at the Neutrino 2016 conference. Each of these analysis campaigns featured different degrees of data intensiveness.

The analysis applications consumed up to 1 GB of input per core hour of analysis and produced 1 GB of physics output. They ran at the scale of 7,500 concurrent cores on the EC2 Spot Market, for a total of over 400,000 core hours, more than doubling their processing capacity during critical weeks of a multi-month processing campaign.

By buffering the input and output data on Amazon S3, NOvA was able to feed data to the analyses at peak bandwidths above 1 GB/s, thus minimizing IO wait and cost. The image above depicts NOvA’s submission system for their data intensive workflows to the AWS infrastructure using HEP Cloud. Given the integration of the data management middleware with Amazon S3, the scientific applications will continue to use the known interfaces for handling massive amounts of data by the experiments.

The large data volume handling was also enabled by the recently upgraded peering point between Amazon in Oregon and the Energy Science Network (ESNet). This peering point provides a 100-Gbps path for data transport between AWS and the national laboratories and was used to transfer more than 100 TB of input and 150 TB of output at bandwidths ranging 5 to 12 Gbps between Fermilab and AWS. With the strengthening of the Data Egress Waiver program for the publicly funded ISP, AWS is becoming an outstanding resource for data-intensive science.

Peter Shanahan (Co-spokesperson of the NOvA experiment) told me:

Our experience with Amazon Web Services shows its potential as a reliable way to meet our peak data processing needs at times of high demand.

I hope that you enjoyed this brief insight into the ways in which AWS is helping to explore the nature of our universe!

— Sanjay Padhi, Ph.D. – AWS Scientific Computing

We launched EC2 Run Command late last year (read my post, New EC2 Run Command – Remote Instance Management at Scale to learn more). This feature was designed to allow developers, system administrators, and other IT professionals to easily and efficiently manage multiple EC2 instances running Windows or Linux. As I explained in my original post,  you can simply choose the desired command, select the desired instances by attributes, tags, or keywords, and then run the command on the selected instances. EC2 Run Command provides access to the output of the command and also retains a log so that you can see which commands were run on which instances. Last month we made EC2 Run Command even more useful by giving you the ability to create, manage, and share command documents with your colleagues or with all AWS users.

Our customers have taken a liking to EC2 Run Command and are making great use of it. Here are a few of the use cases that have been shared with us:

• Create local users and groups.
• Scan for missing Windows updates and install them.
• Install all applicable Windows updates.
• Manage (start, stop, restart) services.
• Install packages and applications.
• Access local log files.

Hybrid and Cross-Cloud Management
Many AWS customers also have some servers on-premises or on another cloud, and have been looking for a single, unified way to manage their hybrid environment at scale. In order to address this very common use case, we are now opening up Run Command to servers running outside of EC2.

We call these external servers Managed Instances. You can install the AWS SSM Agent on your external servers, activate the agent on each server, and then use your existing commands and command documents to manage them (you can also create new documents, of course).

The agent runs on the following operating systems:

• Windows Server (32 and 64 bit) – 2003-2012, including R2 versions (more info).
• Linux (64 bit) – Red Hat Enterprise Linux 7.1+, CentOS 7.1+ (more info).

If you run a virtualized environment using VMware ESXi, Microsoft Hyper-V, KVM or another hypervisor, you can install the agent on the guest operating system(s) as desired.

For simplicity, the agent needs nothing more than the ability to make HTTPS requests to the SSM endpoint in your desired region. These requests can be direct, or can be routed through a proxy or a gateway, as dictated by your network configuration. When the agent makes a request to AWS, it uses an IAM role to access the SSM API. You’ll set up this role when you activate your first set of servers.

The agent sends some identifying information to AWS. This information includes the fully qualified host name, the platform name and version, the agent version, and the server’s IP address. All of these values are stored securely within AWS, and will be deleted if you choose to unregister the server at some point in the future.

Setting up Managed Instances
The setup process is simple and you should be up and running pretty quickly. Here are the steps:

1. Open up the EC2 Console, locate the Commands section, and click on Activations to create your first activation code. As part of this process the Console will prompt you to create the IAM role that I described above:
   
2. Enter a description for the activation, choose a limit (you can activate up to 1000 servers at a time), set an expiration date, and assign a name that will help you to track the Managed Instances in the Console, then click on Create Activation:
   
3. Capture the Activation Code and the Activation ID:
   
4. Install the SSM Agent on the desired servers, and configure it using the values that you saved in the previous step. You simply download the agent, install it, and then enter the values, as detailed in the installation instructions.
5. Return to the console and click on Managed Instances to verify that everything is working as expected:
   

Running Commands on Managed Instances
Now that your instances are managed by AWS, you can run commands on them. For example:
The status of the commands, along with the output, is available from the Console:

To learn more, read Manage Amazon EC2 Instances Remotely.

Available Now
This feature is available now and you can start using it today in all AWS Regions where Run Command is available (see the Run Command page for details). I am looking forward to hearing how you have put it to use in your environment; leave me a comment and let me know how it works out for you!

My colleague Janna Pellegrino shared the guest post below to introduce you to the newest version of our AWS Security Fundamentals Course.

Information security is deeply important to our customers. AWS Security Fundamentals is a free, online course that introduces you to fundamental cloud computing and AWS security concepts, including AWS access control and management, governance, logging, and encryption methods. It also addresses security-related compliance protocols, risk management strategies, and procedures for auditing AWS security infrastructure.

We have significantly updated this course to help our customers. Updates include:

• New content on AWS security services related to encryption, network security, access control and management, and reporting of user access to AWS services.
• Updated information about the AWS Shared Responsibility Model.
• Demos to teach you how to create encrypted root volumes, configure AWS Web Application Firewall (WAF), and create and run AWS Config Rules to evaluate your AWS environment for compliance.
• More robust content around AWS compliance and assurance programs, and AWS services that help enforce governance, compliance, and risk management.
• A short quiz to assess your knowledge on AWS security concepts and services.

This four-hour, self-paced course is aimed at IT business or security professionals interested in cloud security practices and AWS, as well as IT auditors, analysts, and regulators. It is also a recommended prerequisite for our 3-day Security Operations on AWS course.

You can learn about this course and others training resources at AWS Training.

— Janna Pellegrino, AWS Training and Certification

In the last couple of months I have spoken with several Independent Software Vendors (ISVs) about their AWS Marketplace offerings. I’ve heard some impressive success stories and am working to share as many of them as possible.

Last week I spoke with the founders of Ionic Security to learn more about their product and new AWS Marketplace offering (AWS is an equity partner in Ionic). The platform was designed to meet the demanding needs of large-scale customers that operate in regulated industries. It helps them to understand how their data is being used and what actions are going to be performed on the data.

Using the theme of Data Management as a Service, the platform supports a common set of protocols that are accessible via web, desktop, and mobile applications.

Ionic provides a distributed data protection and control environment that allows customers to protect and manage data on a very fine-grained basis, down to individual fields in a document. Each data element is assigned a unique, 64-bit KeyID via a running instance of the Ionic Key Server. Keys can be protected by AWS CloudHSM or by an on-premises Hardware Security Module. In either case, the HSM is responsible for protecting the encryption keys.

Developers use the Ionic SDK to build Ionic-enabled applications. SDK functions support device detection / fingerprinting, key management, encryption & decryption, and logging. The applications make use of an attribute-based policy engine that makes rapid go/no-go decisions when an application wants to access or update a data element. Support for more than 100 document management, productivity, and collaboration applications is already available.

The Ionic Dashboard is the mission control center for the product. It displays all interaction with the data, with multiple forms of visualization, filtering, and drill-down.

Organizations can choose to store the keys and the logging data in several different locations. With today’s announcement, they can now store their data in the AWS region of their choice. The product launches with just a couple of clicks and is up and running within two minutes or so.

Ionic in Action
Here are a couple of screen shots of Ionic in action. The Data Policy Overview provides an overview of activity:

There’s also a geographic view:

Detailed information is available for each data protection policy:

To get started, visit Ionic in AWS Marketplace. You may also want check out the Ionic resources page.

Many applications use the Amazon SNS to deliver SMS messages to individuals or to groups. Because SNS is simple and easy to use, you can add push notifications to your application with a modest amount of code.

Today we are making SNS even more useful by adding support for worldwide delivery of SMS messages. You can now use SNS to send SMS text messages to mobile phone numbers in more than 200 countries.

Along with this change we are also adding several other features to this feature:

• No Opt-In – Message recipients no longer need to opt-in to SNS before they can receive messages.
• Additional Regions – You can now send messages with SMS delivery from the US East (Northern Virginia), US West (Oregon), Europe (Ireland), Asia Pacific (Tokyo), Asia Pacific (Sydney), and Asia Pacific (Singapore) Regions.
• Direct Publishing – You can now send messages to a phone number without first subscribing the number to an SNS topic. If a user replies with “STOP”, they will be added to the opt-out list associated with your AWS account (this feature applies to messages sent to phone numbers located in countries where local regulations require opt-out capabilities).
• Opt-Out Management – You can now manage opt-out numbers using the AWS Management Console, API, or CLI (learn more).
• Delivery Status – You can now enable logging of delivery status for SMS messages and access the status in a pair of CloudWatch Log Groups (Successful and Failed).
• Limit – You can now set spending limits on a per-account, per-month basis. You can also set a per-message spending limit.
• Usage Reports – You can now access daily SMS usage reports, with information on successful and failed deliveries (learn more).
• Long/Short Code Pool – Messages sent via SNS will now appear to come from one of several long or short codes maintained by Amazon SNS. Multiple messages sent from one account to a particular phone number will all be sent from the same long or short code.

To learn more about all of these new features, read Amazon SNS Worldwide SMS Delivery on the AWS Mobile Development Blog.

The portfolio of AWS storage products has grown increasingly rich and diverse over time. Amazon S3 started out with a single storage class and has grown to include storage classes for regular, infrequently accessed, and archived objects. Similarly, Amazon Elastic Block Store (EBS) began with a single volume type and now offers a choice of four types of SAN-style block storage, each designed to be a great for a particular set of access patterns and data types.

With object storage and block storage capably addressed by S3 and EBS, we turned our attention to the file system. We announced the Amazon Elastic File System (EFS) last year in order to provide multiple EC2 instances with shared, low-latency access to a fully-managed file system.

I am happy to announce that EFS is now available for production use in the US East (Northern Virginia), US West (Oregon), and Europe (Ireland) Regions.

We are launching today after an extended preview period that gave us insights into an extraordinarily wide range of customer use cases. The EFS preview was a great fit for large-scale, throughput-heavy processing workloads, along with many forms of content and web serving. During the preview we received a lot of positive feedback about the performance of EFS for these workloads, along with requests to provide equally good support for workloads that are sensitive to latency and/or make heavy use of file system metadata. We’ve been working to address this feedback and today’s launch is designed to handle a very wide range of use cases. Based on what I have heard so far, our customers are really excited about EFS and plan to put it to use right away.

Why We Built EFS
Many AWS customers have asked us for a way to more easily manage file storage on a scalable basis. Some of these customers run farms of web servers or content management systems that benefit from a common namespace and easy access to a corporate or departmental file hierarchy. Others run HPC and Big Data applications that create, process, and then delete many large files, resulting in storage utilization and throughput demands that vary wildly over time. Our customers also insisted on high availability, and durability, along with a strongly consistent model for access and modification.

Amazon Elastic File System
EFS lets you create POSIX-compliant file systems and attach them to one or more of your EC2 instances via NFS. The file system grows and shrinks as necessary (there’s no fixed upper limit and you can grow to petabyte scale) and you don’t pre-provision storage space or bandwidth. You pay only for the storage that you use.

EFS protects your data by storing copies of your files, directories, links, and metadata in multiple Availability Zones.

In order to provide the performance needed to support large file systems accessed by multiple clients simultaneously, Elastic File System performance scales with storage (I’ll say more about this later).

Each Elastic File System is accessible from a single VPC, and is accessed by way of mount targets that you create within the VPC. You have the option to create a mount target in any desired subnet of your VPC. Access to each mount target is controlled, as usual, via Security Groups.

EFS offers two distinct performance modes. The first mode, General Purpose, is the default. You should use this mode unless you expect to have tens, hundreds, or thousands of EC2 instances access the file system concurrently. The second mode, Max I/O, is optimized for higher levels of aggregate throughput and operations per second, but incurs slightly higher latencies for file operations. In most cases, you should start with general purpose mode and watch the relevant CloudWatch metric (PercentIOLimit). When you begin to push the I/O limit of General Purpose mode, you can create a new file system in Max I/O mode, migrate your files, and enjoy even higher throughput and operations per second.

Elastic File System in Action
It is very easy to create, mount, and access an Elastic File System. I used the AWS Management Console; I could have used the EFS API, the AWS Command Line Interface (CLI), or the AWS Tools for Windows PowerShell as well.

I opened the console and clicked on the Create file system button:

Then I selected one of my VPCs and created a mount target in my public subnet:

My security group (corp-vpc-mount-target) allows my EC2 instance to access the mount point on port 2049. Here’s the inbound rule; the outbound one is the same:

I added Name and Owner tags, and opted for the General Purpose performance mode:

Then I confirmed the information and clicked on Create File System:

My file system was ready right away (the mount targets took another minute or so):

I clicked on EC2 mount instructions to learn how to mount my file system on an EC2 instance:

I mounted my file system as /efs, and there it was:

I copied a bunch of files over, and spent some time watching the NFS stats:

The console reports on the amount of space consumed by my file systems (this information is collected every hour and is displayed 2-3 hours after it is collected):

CloudWatch Metrics
Each file system delivers the following metrics to CloudWatch:

• BurstCreditBalance – The amount of data that can be transferred at the burst level of throughput.
• ClientConnections – The number of clients that are connected to the file system.
• DataReadIOBytes – The number of bytes read from the file system.
• DataWriteIOBytes -The number of bytes written to the file system.
• MetadataIOBytes – The number of bytes of metadata read and written.
• TotalIOBytes -The sum of the preceding three metrics.
• PermittedThroughput -The maximum allowed throughput, based on file system size.
• PercentIOLimit – The percentage of the available I/O utilized in General Purpose mode.

You can see the metrics in the CloudWatch Console:

EFS Bursting, Workloads, and Performance
The throughput available to each of your EFS file systems will grow as the file system grows. Because file-based workloads are generally spiky, with demands for high levels of throughput for short amounts of time and low levels the rest of the time, EFS is designed to burst to high throughput levels on an as-needed basis.

All file systems can burst to 100 MB per second of throughput. Those over 1 TB can burst to an additional 100 MB per second for each TB stored. For example, a 2 TB file system can burst to 200 MB per second and a 10 TB file system can burst to 1,000 MB per second of throughput. File systems larger than 1 TB can always burst for 50% of the time if they are inactive for the other 50%.

EFS uses a credit system to determine when a file system can burst. Each one accumulates credits at a baseline rate (50 MB per TB of storage) that is determined by the size of the file system, and spends them whenever it reads or writes data. The accumulated credits give the file system the ability to drive throughput beyond the baseline rate.

Here are some examples to give you a better idea of what this means in practice:

• A 100 GB file system can burst up to 100 MB per second for up to 72 minutes each day, or drive up to 5 MB per second continuously.
• A 10 TB file system can burst up to 1 GB per second for 12 hours each day, or drive 500 MB per second continuously.

To learn more about how the credit system works, read about File System Performance in the EFS documentation.

In order to gain a better understanding of this feature, I spent a couple of days copying and concatenating files, ultimately ending up using well over 2 TB of space on my file system. I watched the PermittedThroughput metric grow in concert with my usage as soon as my file collection exceeed 1 TB. Here’s what I saw:

As is the case with any file system, the throughput you’ll see is dependent on the characteristics of your workload. The average I/O size, the number of simultaneous connections to EFS, the file access pattern (random or sequential), the request model (synchronous or asynchronous), the NFS client configuration, and the performance characteristics of the EC2 instances running the NFS clients each have an effect (positive or negative). Briefly:

• Average I/O Size – The work associated with managing the metadata associated with small files via the NFS protocol, coupled with the work that EFS does to make your data highly durable and highly available, combine to create some per-operation overhead. In general, overall throughput will increase in concert with the average I/O size since the per-operation overhead is amortized over a larger amount of data. Also, reads will generally be faster than writes.
• Simultaneous Connections – Each EFS file system can accommodate connections from thousands of clients. Environments that can drive highly parallel behavior (from multiple EC2 instances) will benefit from the ability that EFS has to support a multitude of concurrent operations.
• Request Model – If you enable asynchronous writes to the file system by including the async option at mount time, pending writes will be buffered on the instance and then written to EFS asynchronously. Accessing a file system that has been mounted with the sync option or opening files using an option that bypasses the cache (e.g. O_DIRECT) will, in turn, issue synchronous requests to EFS.
• NFS Client Configuration – Some NFS clients use laughably small (by today’s standards) values for the read and write buffers by default. Consider increasing it to 1 MiB (again, this is an option to the mount command). You can use an NFS 4.0 or 4.1 client with EFS; the latter will provide better performance.
• EC2 Instances – Applications that perform large amounts of I/O sometimes require a large amount of memory and/or compute power as well. Be sure that you have plenty of both; choose an appropriate instance size and type. If you are performing asynchronous reads and writes, the kernel use additional memory for caching. As a side note, the performance characteristics of EFS file systems are not dependent on the use of EBS-optimized instances.

Benchmarking of file systems is a blend of art and science. Make sure that you use mature, reputable tools, run them more than once, and make sure that you examine your results in light of the considerations listed above. You can also find some detailed data regarding expected performance on the Amazon Elastic File System page.

Available Now
EFS is available now in the US East (Northern Virginia), US West (Oregon), and Europe (Ireland) Regions and you can start using it today.  Pricing is based on the amount of data that you store, sampled several times per day and charged by the Gigabyte-month, pro-rated as usual, starting at $0.30 per GB per month in the US East (Northern Virginia) Region. There are no minimum fees and no setup costs (see the EFS Pricing page for more information). If you are eligible for the AWS Free Tier, you can use up to 5 GB of EFS storage per month at no charge.

— Jeff;

Many AWS customers create high-performance systems that run across multiple EC2 instances and make good use of all available network bandwidth. Over the years, we have been working to make EC2 an ever-better host for this use case. For example, the CC1 instances were the first to feature 10 Gbps networking. Later, the Enhanced Networking feature reduced latency, increased the packet rate, and reduced variability. Through the years, the goal has remained constant—ensure that the network is not the bottleneck for the vast majority of customer workloads.

We’ve been talking to lots of customers and watching technology trends closely. Along with a short-term goal of enabling higher throughput by providing more bandwidth, we established some longer-term goals for the next generation of EC2 networking. We wanted to be able to take advantage of the increased concurrency (more vCPUs) found in today’s processors, and we wanted to lay a solid foundation for driver support in order to allow our customers to take advantage of new developments as easily as possible.

I’m happy to be able to report that we are making great progress toward these goals with today’s launch of the new Elastic Network Adapter (ENA) to provide even better support for high performance networking. Available now for the new X1 instance type, ENA provides up to 20 Gbps of consistent, low-latency performance when used within a Placement Group, at no extra charge!

Per our longer-term goals, ENA will scale as network bandwidth grows and the vCPU count increases; this will allow you to take advantage of higher bandwidth options in the future without the need to install newer drivers or to make other changes to your configuration, as was required by earlier network interfaces.

ENA Advantages
We designed ENA to work well in conjunction with modern processors, such as those found on X1 instances. Because these processors feature a large number of virtual CPUs (128 in the case of  X1), efficient use of shared resources such as the network adapter is important. While delivering high throughput and great packet per second (PPS) performance, ENA minimizes the load on the host processor in a number of ways, and also does a better job of distributing the packet processing workload across multiple vCPUs. Here are some of the features that enable this improved performance:

• Checksum Generation – ENA handles IPv4 header checksum generation and TCP / UDP partial checksum generation in hardware.
• Multi-Queue Device Interface – ENA makes uses of multiple transmit and receive queues to reduce internal overhead and to increase scalability. The presence of multiple queues simplifies and accelerates the process of mapping incoming and outgoing packets to a particular vCPU.
• Receive-Side Steering – ENA is able to direct incoming packets to the proper vCPU for processing. This reduces bottlenecks and increases cache efficacy.

All of these features are designed to keep as much of the workload off of the processor as possible and to create a short, efficient path between the network packets and the vCPU that is generating or processing them.

Using ENA
In order to make use of ENA,  you need to use our new driver and tag the AMI as having ENA support.

The new driver is available in the latest Amazon Linux AMIs and will soon be available in the Windows AMIs. The Open Source Linux driver is available in source form on GitHub for use in your own AMIs. Also, a driver for the Intel^® Data Plane Developer Kit (Intel^® DPDK) is available for developers that are building network processing applications such as load balancers or virtual routers.

If you are creating your own AMI, you also need to set the enaSupport attribute when you register it. Here’s how you do that from the command line (see the register-image documentation for a full list of parameters):

$ aws ec2 register-image --ena-support ...

You can still use the AMI on instances that do not support ENA.

Going Forward
As noted earlier, ENA is available today for X1 instances. We are also planning to make it available for future EC2 instance types.