AWS News Blog

This guest post is by Colm MacCárthaigh, Senior Engineer for Amazon Virtual Private Cloud.

Since VPC Endpoints launched in 2015, creating Endpoints has been a popular way to securely access S3 and DynamoDB from an Amazon Virtual Private Cloud (VPC) without the need for an Internet gateway, a NAT gateway, or firewall proxies. With VPC Endpoints, the routing between the VPC and the AWS service is handled by the AWS network, and IAM policies can be used to control access to service resources.

Today we are announcing AWS PrivateLink, the newest generation of VPC Endpoints which is designed for customers to access AWS services in a highly available and scalable manner, while keeping all the traffic within the AWS network. Kinesis, Service Catalog, Amazon EC2, EC2 Systems Manager (SSM), and Elastic Load Balancing (ELB) APIs are now available to use inside your VPC, with support for more services coming soon such as Key Management Service (KMS) and Amazon Cloudwatch.

With traditional endpoints, it’s very much like connecting a virtual cable between your VPC and the AWS service. Connectivity to the AWS service does not require an Internet or NAT gateway, but the endpoint remains outside of your VPC. With PrivateLink, endpoints are instead created directly inside of your VPC, using Elastic Network Interfaces (ENIs) and IP addresses in your VPC’s subnets. The service is now in your VPC, enabling connectivity to AWS services via private IP addresses. That means that VPC Security Groups can be used to manage access to the endpoints and that PrivateLink endpoints can also be accessed from your premises via AWS Direct Connect.

Using the services powered by PrivateLink, customers can now manage fleets of instances, create and manage catalogs of IT services as well as store and process data, without requiring the traffic to traverse the Internet.

Creating a PrivateLink Endpoint
To create a PrivateLink endpoint, I navigate to the VPC Console, select Endpoints, and choose Create Endpoint.

I then choose which service I’d like to access. New PrivateLink endpoints have an “interface” type. In this case I’d like to use the Kinesis service directly from my VPC and I choose the kinesis-streams service.

At this point I can choose which of my VPCs I’d like to launch my new endpoint in, and select the subnets that the ENIs and IP addresses will be placed in. I can also associate the endpoint with a new or existing Security Group, allowing me to control which of my instances can access the Endpoint.

Because PrivateLink endpoints will use IP addresses from my VPC, I have the option to over-ride DNS for the AWS service DNS name by using VPC Private DNS. By leaving Enable Private DNS Name checked, lookups from within my VPC for “kinesis.us-east-1.amazonaws.com” will resolve to the IP addresses for the endpoint that I’m creating. This makes the transition to the endpoint seamless without requiring any changes to my applications. If I’d prefer to test or configure the endpoint before handling traffic by default, I can leave this disabled and then change it at any time by editing the endpoint.

Once I’m ready and happy with the VPC, subnets and DNS settings, I click Create Endpoint to complete the process.

Using a PrivateLink Endpoint

By default, with the Private DNS Name enabled, using a PrivateLink endpoint is as straight-forward as using the SDK, AWS CLI or other software that accesses the service API from within your VPC. There’s no need to change any code or configurations.

To support testing and advanced configurations, every endpoint also gets a set of DNS names that are unique and dedicated to your endpoint. There’s a primary name for the endpoint and zonal names.

The primary name is particularly useful for accessing your endpoint via Direct Connect, without having to use any DNS over-rides on-premises. Naturally, the primary name can also be used inside of your VPC.
The primary name, and the main service name – since I chose to over-ride it – include zonal fault-tolerance and will balance traffic between the Availability Zones. If I had an architecture that uses zonal isolation techniques, either for fault containment and compartmentalization, low latency, or for minimizing regional data transfer I could also use the zonal names to explicitly control whether my traffic flows between or stays within zones.

Pricing & Availability
AWS PrivateLink is available today in all AWS commercial regions except China (Beijing). For the region availability of individual services, please check our documentation.

Pricing starts at $0.01 / hour plus a data processing charge at $0.01 / GB. Data transferred between availability zones, or between your Endpoint and your premises via Direct Connect will also incur the usual EC2 Regional and Direct Connect data transfer charges. For more information, see VPC Pricing.

–Colm MacCárthaigh

Back in 2006, when I announced S3, I wrote ” Further, each block is protected by an ACL (Access Control List) allowing the developer to keep the data private, share it for reading, or share it for reading and writing, as desired.”

Starting from that initial model, with private buckets and ACLs to grant access, we have added support for bucket policies, server access logging, versioning, API logging, cross-region replication, and multiple client-side and server-side encryption options, all with the goal of giving you the tools you need to keep your data safe while allowing you to share it with customers and partners as needed. We also brought the power of Artificial Intelligence and machine learning in to play, with the launch of Amazon Macie, a tool that helps you to discover, classify, and secure content at scale.

Today we are adding five new encryption and security features to S3:

Default Encryption – You can now mandate that all objects in a bucket must be stored in encrypted form without having to construct a bucket policy that rejects objects that are not encrypted.

Permission Checks – The S3 Console now displays a prominent indicator next to each S3 bucket that is publicly accessible.

Cross-Region Replication ACL Overwrite – When you replicate objects across AWS accounts, you can now specify that the object gets a new ACL that gives full access to the destination account.

Cross-Region Replication with KMS – You can now replicate objects that are encrypted with keys that are managed by AWS Key Management Service (KMS).

Detailed Inventory Report – The S3 Inventory report now includes the encryption status of each object. The report itself can also be encrypted.

Let’s take a look at each one…

Default Encryption
You have three server-side encryption options for your S3 objects: SSE-S3 with keys that are managed by S3, SSE-KMS with keys that are managed by AWS KMS, and SSE-C with keys that you manage. Some of our customers, particularly those who need to meet compliance requirements that dictate the use of encryption at rest, have used bucket policies to ensure that every newly stored object is encrypted. While this helps them to meet their requirements, simply rejecting the storage of unencrypted objects is an imperfect solution.

You can now mandate that all objects in a bucket must be stored in encrypted form by installing a bucket encryption configuration. If an unencrypted object is presented to S3 and the configuration indicates that encryption must be used, the object will be encrypted using encryption option specified for the bucket (the PUT request can also specify a different option).

Here’s how to enable this feature using the S3 Console when you create a new bucket. Enter the name of the bucket as usual and click on Next. Then scroll down and click on Default encryption:

Select the desired option and click on Save (if you select AWS-KMS you also get to designate a KMS key):

You can also make this change via a call to the PUT Bucket Encryption function. It must specify an SSE algorithm (either SSE-S3 or SSE-KMS) and can optionally reference a KMS key.

Keep the following in mind when you implement this feature:

SigV4 – Access to the bucket policy via the S3 REST API must be signed with SigV4 and made over an SSL connection.

Updating Bucket Policies – You should examine and then carefully modify existing bucket policies that currently reject unencrypted objects.

High-Volume Use – If you are using SSE-KMS and are uploading many hundreds or thousands of objects per second, you may bump in to the KMS Limit on the Encrypt and Decrypt operations. Simply file a Support Case and ask for a higher limit:

Cross-Region Replication – Unencrypted objects will be encrypted according to the configuration of the destination bucket. Encrypted objects will remain as such.

Permission Checks
The combination of bucket policies, bucket ACLs, and Object ACLs give you very fine-grained control over access to your buckets and the objects within them. With the goal of making sure that your policies and ACLs combine to create the desired effect, we recently launched a set of Managed Config Rules to Secure Your S3 Buckets. As I mentioned in the post, these rules make use of some of our work to put Automated Formal Reasoning to use.

We are now using the same underlying technology to help you see the impact of changes to your bucket policies and ACLs as soon as you make them. You will know right away if you open up a bucket for public access, allowing you to make changes with confidence.

Here’s what it looks like on the main page of the S3 Console (I’ve sorted by the Access column for convenience):

The Public indicator is also displayed when you look inside a single bucket:

You can also see which Permission elements (ACL, Bucket Policy, or both) are enabling the public access:

Cross-Region Replication ACL Overwrite
Our customers often use S3’s Cross-Region Replication to copy their mission-critical objects and data to a destination bucket in a separate AWS account. In addition to copying the object, the replication process copies the object ACL and any tags associated with the object.

We’re making this feature even more useful by allowing you to enable replacement of the ACL as it is in transit so that it grants full access to the owner of the destination bucket. With this change, ownership of the source and the destination data is split across AWS accounts, allowing you to maintain separate and distinct stacks of ownership for the original objects and their replicas.

To enable this feature when you are setting up replication, choose a destination bucket in a different account and Region by specifying the Account ID and Bucket name, and clicking on Save:

Then click on Change object ownership…:

We’ve also made it easier for you to set up the key policy for the destination bucket in the destination account. Simply log in to the account and find the bucket, then click on Management and Replication, then choose Receive objects… from the More menu:

Enter the source Account ID, enable versioning, inspect the policies, apply them, and click on Done:

Cross-Region Replication with KMS
Replicating objects that have been encrypted using SSE-KMS across regions poses an interesting challenge that we are addressing today. Because the KMS keys are specific to a particular region, simply replicating the encrypted object would not work.

You can now choose the destination key when you set up cross-region replication. During the replication process, encrypted objects are replicated to the destination over an SSL connection. At the destination, the data key is encrypted with the KMS master key you specified in the replication configuration. The object remains in its original, encrypted form throughout; only the envelope containing the keys is actually changed.

Here’s how you enable this feature when you set up a replication rule:

As I mentioned earlier, you may need to request an increase in your KMS limits before you start to make heavy use of this feature.

Detailed Inventory Report
Last but not least, you can now request that the daily or weekly S3 inventory reports include information on the encryption status of each object:

As you can see, you can also request SSE-S3 or SSE-KMS encryption for the report.

Now Available
All of these features are available now and you can start using them today! There is no charge for the features, but you will be charged the usual rates for calls to KMS, S3 storage, S3 requests, and inter-region data transfer.

— Jeff;

We launched Convertible Reserved Instances for EC2 just about a year ago. The Convertible RIs give you a significant discount (typically 54% when compared to On-Demand) and allow you to change the instance family and other parameters associated with the RI if your needs change.

Today we are introducing Convertible RIs with a 1-year term, complementing the existing 3-year term. We are also making the Convertible Reserved Instance model more flexible by allowing you to exchange portions of your RIs and to perform bulk exchanges.

New 1-Year Convertible RIs
Convertible Reserved Instances with a 1-year term are now available. This will give you more options and more flexibility; you can now purchase a mix of 1-year and 3-year Convertible Reserved Instances (CRIs) in accord with your needs. Startups with financial constraints will find this option attractive, as will other ventures that may not be in a position to make a commitment that runs for longer than one year.

Merging and Splitting Convertible RIs
Let’s say that you start running your web and application servers on M4 instances and uses Convertible RIs to save money. Later, after a tuning exercise you move your application servers to C4 instances. With today’s launch you can exchange a portion of your M4 Convertible RIs for C4 Convertible RIs. You can also merge two or more CRIs (perhaps for smaller instances) and obtain one for a larger instance.

The exchange model for Convertible Reserved Instances is based on splitting, exchanging, and merging. Let’s say I own a 3-year Partial Upfront CRI for four t2.micro instances:

My application has changed and now I want to use a pair of t2.micro instances and a single r4.xlarge. The first step is to split this CRI into the part that I want to keep and the part that I want to exchange. I select it and click on Modify Reserved Instances. Then I create my desired configuration and click on Continue:

I review the request and click on Submit Modifications:

The state of the CRI changes to indicate that it is being modified. After a moment or two it will be marked as retired, replaced by a pair that are active:

Now I can exchange one of the 2-instance CRIs. I select it, click on Exchange Reserved Instance, and enter the desired configuration for my new CRI:

I click on Find Offering to see my options, and choose the desired one, an r4.xlarge Partial Upfront. As you can see, the console “does the math” takes the remaining upfront value ($139.995 in this case) of the unneeded CRIs into account when computing the upfront payment:

When I am ready to move forward I click on Exchange. This initiates the exchange process and lets me know that it may take a few minutes to complete.

I can also merge two or more Convertible Reserved Instances together and then use them as the starting point for an exchange. To do this I simply select the existing CRIs, click on Action, and choose Exchange Reserved Instances. I can see the total remaining upfront value of the selected CRIs and proceed accordingly:

You can merge CRIs that have different start dates and/or term lengths. The merged CRI will have the expiry date of the RI that is furthest from the date of exchange. Merging CRIs with different term lengths always produces a 3-year CRI.

You can also perform the split, exchange, and merge operations using the AWS Command Line Interface (CLI) and the EC2 APIs.

Available Now
All of the functions and the 1-year CRIs described in this post are available now and you can start using them today.

— Jeff;

The original AWS Price List API, as described in New – AWS Price List API, gave you access to prices in JSON and CSV form by way of structured URLs. While this worked well for some types of cost management tools, the size and complexity of the files made them difficult to download and tedious to parse. Today we are updating the API by adding new functions that allow you to perform fine-grained price queries that return only the prices that you need. This will allow you to make use of the prices in mobile and browser-based applications.

New Functions
Here are the new functions:

DescribeServices – Returns sets of attribute keys that are used to define products within a service. For example, the keys returned for EC2 will include physicalProcessor, memory, operatingSystem, location, and tenancy.

GetAttributeValues – Returns all of the allowable values for a given attribute key. For example, values for the operatingSystem key include Windows, RHEL, Linux, and SUSE; values for the location key include US East (N. Virginia) and Asia Pacific (Mumbai).

GetProducts -Returns all of the products, along with their public prices, that match a filter expression based on service name and attribute value.

You can access these functions from the AWS SDKs. In order to try them out I used Python and the AWS SDK for Python. I start by importing the SDK and creating a client:

import boto3
import json
import pprint

pricing = boto3.client('pricing')

Here’s how I list all of the services and attributes:

print("All Services")
print("============")
response = pricing.describe_services()
for service in response['Services']:
    print(service['ServiceCode'] + ": " + ", ".join(service['AttributeNames']))
print()

The output starts like this:

All Services
============
SnowballExtraDays: productFamily, termType, usagetype, locationType, snowballType, feeDescription, servicecode, feeCode, location, operation
OpsWorks: productFamily, servicecode, termType, usagetype, locationType, location, operation, serverLocation, group
mobileanalytics: productFamily, servicecode, includedEvents, termType, usagetype, description, locationType, location, operation
IngestionServiceSnowball: productFamily, fromLocationType, termType, usagetype, locationType, toLocationType, toLocation, snowballType, servicecode, groupDescription, transferType, location, fromLocation, operation, group
IngestionService: productFamily, termType, usagetype, locationType, servicecode, groupDescription, dataAction, location, operation, group
ElasticMapReduce: productFamily, softwareType, instanceType, termType, usagetype, locationType, instanceFamily, servicecode, location, servicename, operation
datapipeline: productFamily, frequencyMode, termType, usagetype, locationType, description, executionFrequency, servicecode, location, operation, group, executionLocation
...

Here’s how I get all of the values for all of EC2’s pricing attributes:

print("Selected EC2 Attributes & Values")
print("================================")
response = pricing.describe_services(ServiceCode='AmazonEC2')
attrs = response['Services'][0]['AttributeNames']

for attr in attrs:
    response = pricing.get_attribute_values(ServiceCode='AmazonEC2', AttributeName=attr)

    values = []
    for attr_value in response['AttributeValues']:
        values.append(attr_value['Value'])

    print("  " + attr + ": " + ", ".join(values))

The output starts like this:

Selected EC2 Attributes & Values
================================
  volumeType: Throughput Optimized HDD, Provisioned IOPS, Magnetic, General Purpose, Cold HDD
  maxIopsvolume: 500 - based on 1 MiB I/O size, 40 - 200, 250 - based on 1 MiB I/O size, 20000, 10000
  instanceCapacity10xlarge: 1
  locationType: AWS Region
  instanceFamily: Storage optimized, Micro instances, Memory optimized, GPU instance, General purpose, Compute optimized
  operatingSystem: Windows, SUSE, RHEL, NA, Linux
...

And here’s how I use the service name and attribute values to obtain price listings for EC2 instances with 64 vCPUs, 256 GiB of memory, pre-installed SQL Server Enterprise, in the Asia Pacific (Mumbai) Region. Each price is a JSON string:

print("Selected EC2 Products")
print("=====================")

response = pricing.get_products(
     ServiceCode='AmazonEC2',
     Filters = [
         {'Type' :'TERM_MATCH', 'Field':'operatingSystem', 'Value':'Windows'              },
         {'Type' :'TERM_MATCH', 'Field':'vcpu',            'Value':'64'                   },
         {'Type' :'TERM_MATCH', 'Field':'memory',          'Value':'256 GiB'              },
         {'Type' :'TERM_MATCH', 'Field':'preInstalledSw',  'Value':'SQL Ent'              },
         {'Type' :'TERM_MATCH', 'Field':'location',        'Value':'Asia Pacific (Mumbai)'}
     ],
     MaxResults=100
)

for price in response['PriceList']:
 pp = pprint.PrettyPrinter(indent=1. width=300)
 pp.pprint(json.loads(price))
 print()

The output starts like this (there’s a lot more):

Selected EC2 Products
=====================
{'product': {'attributes': {'clockSpeed': '2.3 GHz',
                            'currentGeneration': 'Yes',
                            'dedicatedEbsThroughput': '10000 Mbps',
                            'ecu': '188',
                            'enhancedNetworkingSupported': 'Yes',
                            'instanceFamily': 'General purpose',
                            'instanceType': 'm4.16xlarge',
                            'licenseModel': 'No License required',
                            'location': 'Asia Pacific (Mumbai)',
                            'locationType': 'AWS Region',
                            'memory': '256 GiB',
                            'networkPerformance': '20 Gigabit',
                            'normalizationSizeFactor': '128',
                            'operatingSystem': 'Windows',
                            'operation': 'RunInstances:0102',
                            'physicalProcessor': 'Intel Xeon E5-2686 v4 (Broadwell)',
                            'preInstalledSw': 'SQL Ent',
                            'processorArchitecture': '64-bit',
                            'processorFeatures': 'Intel AVX, Intel AVX2, Intel Turbo',
                            'servicecode': 'AmazonEC2',
                            'servicename': 'Amazon Elastic Compute Cloud',
                            'storage': 'EBS only',
                            'tenancy': 'Shared',
                            'usagetype': 'APS3-BoxUsage:m4.16xlarge',
                            'vcpu': '64'},
             'productFamily': 'Compute Instance',
             'sku': '24GRA8RB2KZ9NPCS'},
 'publicationDate': '2017-10-07T00:26:55Z',
 'serviceCode': 'AmazonEC2',
...

The next part of the response contains an array of terms, each describing a particular way to purchase the instance (On-Demand or various types of Reserved Instance):

 'terms': {'OnDemand': {'24GRA8RB2KZ9NPCS.JRTCKXETXF': {'effectiveDate': '2017-09-01T00:00:00Z',
                                                        'offerTermCode': 'JRTCKXETXF',
                                                        'priceDimensions': {'24GRA8RB2KZ9NPCS.JRTCKXETXF.6YS6EN2CT7': {'appliesTo': [],
                                                                                                                       'beginRange': '0',
                                                                                                                       'description': '$30.88 per On Demand Windows with SQL Server Enterprise m4.16xlarge Instance Hour',
                                                                                                                       'endRange': 'Inf',
                                                                                                                       'pricePerUnit': {'USD': '30.8800000000'},
                                                                                                                       'rateCode': '24GRA8RB2KZ9NPCS.JRTCKXETXF.6YS6EN2CT7',
                                                                                                                       'unit': 'Hrs'}},
                                                        'sku': '24GRA8RB2KZ9NPCS',
                                                        'termAttributes': {}}},
           'Reserved': {'24GRA8RB2KZ9NPCS.38NPMPTW36': {'effectiveDate': '2017-04-30T23:59:59Z',
                                                        'offerTermCode': '38NPMPTW36',
                                                        'priceDimensions': {'24GRA8RB2KZ9NPCS.38NPMPTW36.2TG2D8R56U': {'appliesTo': [], 'description': 'Upfront Fee', 'pricePerUnit': {'USD': '374227'}, 'rateCode': '24GRA8RB2KZ9NPCS.38NPMPTW36.2TG2D8R56U', 'unit': 'Quantity'},
                                                                            '24GRA8RB2KZ9NPCS.38NPMPTW36.6YS6EN2CT7': {'appliesTo': [],
                                                                                                                       'beginRange': '0',
                                                                                                                       'description': 'Windows with SQL Server Enterprise (Amazon VPC), m4.16xlarge reserved instance applied',
                                                                                                                       'endRange': 'Inf',
                                                                                                                       'pricePerUnit': {'USD': '14.2400000000'},
                                                                                                                       'rateCode': '24GRA8RB2KZ9NPCS.38NPMPTW36.6YS6EN2CT7',
                                                                                                                       'unit': 'Hrs'}},
                                                        'sku': '24GRA8RB2KZ9NPCS',
                                                        'termAttributes': {'LeaseContractLength': '3yr', 'OfferingClass': 'standard', 'PurchaseOption': 'Partial Upfront'}},
...

Read Using the AWS Price List API to learn more about these functions and the data that they return.

Now Available
The new functions are available now and you can start using them today in the US East (Northern Virginia) and Asia Pacific (Mumbai) Regions to access metadata and price listings for all public AWS Regions and AWS GovCloud (US), at no charge.

To see a real-world example of how to use these functions, take a look at the new post, Controlling Projected User Costs Through Monthly Budget Policies on the AWS Management Tools Blog.

— Jeff;

I’m thrilled to announce that the new compute-intensive C5 instances are available today in six sizes for launch in three AWS regions!

These instances are designed for compute-heavy applications like batch processing, distributed analytics, high-performance computing (HPC), ad serving, highly scalable multiplayer gaming, and video encoding. The new instances offer a 25% price/performance improvement over the C4 instances, with over 50% for some workloads. They also have additional memory per vCPU, and (for code that can make use of the new AVX-512 instructions), twice the performance for vector and floating point workloads.

Over the years we have been working non-stop to provide our customers with the best possible networking, storage, and compute performance, with a long-term focus on offloading many types of work to dedicated hardware designed and built by AWS. The C5 instance type incorporates the latest generation of our hardware offloads, and also takes another big step forward with the addition of a new hypervisor that runs hand-in-glove with our hardware. The new hypervisor allows us to give you access to all of the processing power provided by the host hardware, while also making performance even more consistent and further raising the bar on security. We’ll be sharing many technical details about it at AWS re:Invent.

The New Instances
The C5 instances are available in six sizes:

Each vCPU is a hardware hyperthread on a 3.0 GHz Intel Xeon Platinum 8000-series processor. This custom processor, optimized for EC2, allows you have full control over the C-states on the two largest sizes, allowing you to run a single core at up to 3.5 GHz using Intel Turbo Boost Technology.

As you can see from the table, the four smallest instance sizes offer substantially more EBS and network bandwidth than the previous generation of compute-intensive instances.

Because all networking and storage functionality is implemented in hardware, C5 instances require HVM AMIs that include drivers for the Elastic Network Adapter (ENA) and NVMe. The latest Amazon Linux, Microsoft Windows (Windows Server 2012 R2 and Windows Server 2016), Ubuntu, RHEL, CentOS, SLES, Debian, and FreeBSD AMIs all support C5 instances. If you are doing machine learning inferencing, or other compute-intensive work, be sure to check out the most recent version of the Intel Math Kernel Library. It has been optimized for the Intel® Xeon® Platinum processor and has the potential to greatly accelerate your work.

In order to remain compatible with instances that use the Xen hypervisor, the device names for EBS volumes will continue to use the existing /dev/sd and /dev/xvd prefixes. The device name that you provide when you attach a volume to an instance is not used because the NVMe driver assigns its own device name (read Amazon EBS and NVMe to learn more):

The nvme command displays additional information about each volume (install it using sudo yum -y install nvme-cli if necessary):

The SN field in the output can be mapped to an EBS volume ID by inserting a “-” after the “vol” prefix (sadly, the NVMe SN field is not long enough to store the entire ID). Here’s a simple script that uses this information to create an EBS snapshot of each attached volume:

$ sudo nvme list | \
  awk '/dev/ {print(gensub("vol", "vol-", 1, $2))}' | \
  xargs -n 1 aws ec2 create-snapshot --volume-id

With a little more work (and a lot of testing), you could create a script that expands EBS volumes that are getting full.

Getting to C5
As I mentioned earlier, our effort to offload work to hardware accelerators has been underway for quite some time. Here’s a recap:

CC1 – Launched in 2010, the CC1 was designed to support scale-out HPC applications. It was the first EC2 instance to support 10 Gbps networking and one of the first to support HVM virtualization. The network fabric that we designed for the CC1 (based on our own switch hardware) has become the standard for all AWS data centers.

C3 – Launched in 2013, the C3 introduced Enhanced Networking and uses dedicated hardware accelerators to support the software defined network inside of each Virtual Private Cloud (VPC). Hardware virtualization removes the I/O stack from the hypervisor in favor of direct access by the guest OS, resulting in higher performance and reduced variability.

C4 – Launched in 2015, the C4 instances are EBS Optimized by default via a dedicated network connection, and also offload EBS processing (including CPU-intensive crypto operations for encrypted EBS volumes) to a hardware accelerator.

C5 – Launched today, the hypervisor that powers the C5 instances allow practically all of the resources of the host CPU to be devoted to customer instances. The ENA networking and the NVMe interface to EBS are both powered by hardware accelerators. The instances do not require (or support) the Xen paravirtual networking or block device drivers, both of which have been removed in order to increase efficiency.

Going forward, we’ll use this hypervisor to power other instance types and plan to share additional technical details in a set of AWS re:Invent sessions.

Launch a C5 Today
You can launch C5 instances today in the US East (Northern Virginia), US West (Oregon), and EU (Ireland) Regions in On-Demand and Spot form (Reserved Instances are also available), with additional Regions in the works.

One quick note before I go: The current NVMe driver is not optimized for high-performance sequential workloads and we don’t recommend the use of C5 instances in conjunction with sc1 or st1 volumes. We are aware of this issue and have been working to optimize the driver for this important use case.

— Jeff;

So many launches and cloud innovations, that you simply may not believe.  In order to catch up on some service launches and features, this post will be a round-up of some cool releases that happened this summer and through the end of September.

The launches and features I want to share with you today are:

• AWS IAM for Authenticating Database Users for RDS MySQL and Amazon Aurora
• Amazon SES Reputation Dashboard
• Amazon SES Open and Click Tracking Metrics
• Serverless Image Handler by the Solutions Builder Team
• AWS Ops Automator by the Solutions Builder Team

Let’s dive in, shall we!

AWS IAM for Authenticating Database Users for RDS MySQL and Amazon Aurora

Wished you could manage access to your Amazon RDS database instances and clusters using AWS IAM? Well, wish no longer. Amazon RDS has launched the ability for you to use IAM to manage database access for Amazon RDS for MySQL and Amazon Aurora DB.

What I like most about this new service feature is, it’s very easy to get started.  To enable database user authentication using IAM, you would select a checkbox Enable IAM DB Authentication when creating, modifying, or restoring your DB instance or cluster. You can enable IAM access using the RDS console, the AWS CLI, and/or the Amazon RDS API.

After configuring the database for IAM authentication, client applications authenticate to the database engine by providing temporary security credentials generated by the IAM Security Token Service. These credentials can be used instead of providing a password to the database engine.

You can learn more about using IAM to provide targeted permissions and authentication to MySQL and Aurora by reviewing the Amazon RDS user guide.

Amazon SES Reputation Dashboard

In order to aid Amazon Simple Email Service customers’ in utilizing best practice guidelines for sending email, I am thrilled to announce we launched the Reputation Dashboard to provide comprehensive reporting on email sending health. To aid in proactively managing emails being sent, customers now have visibility into overall account health, sending metrics, and compliance or enforcement status.

The Reputation Dashboard will provide the following information:

• Account status: A description of your account health status.
  • Healthy – No issues currently impacting your account.
  • Probation – Account is on probation; Issues causing probation must be resolved to prevent suspension
  • Pending end of probation decision – Your account is on probation. Amazon SES team member must review your account prior to action.
  • Shutdown – Your account has been shut down. No email will be able to be sent using Amazon SES.
  • Pending shutdown – Your account is on probation and issues causing probation are unresolved.
• Bounce Rate: Percentage of emails sent that have bounced and bounce rate status messages.
• Complaint Rate: Percentage of emails sent that recipients have reported as spam and complaint rate status messages.
• Notifications: Messages about other account reputation issues.

Amazon SES Open and Click Tracking Metrics

Another exciting feature recently added to Amazon SES is support for Email Open and Click Tracking Metrics. With Email Open and Click Tracking Metrics feature, SES customers can now track when email they’ve sent has been opened and track when links within the email have been clicked.  Using this SES feature will allow you to better track email campaign engagement and effectiveness.

How does this work?

When using the email open tracking feature, SES will add a transparent, miniature image into the emails that you choose to track. When the email is opened, the mail application client will load the aforementioned tracking which triggers an open track event with Amazon SES. For the email click (link) tracking, links in email and/or email templates are replaced with a custom link.  When the custom link is clicked, a click event is recorded in SES and the custom link will redirect the email user to the link destination of the original email.

You can take advantage of the new open tracking and click tracking features by creating a new configuration set or altering an existing configuration set within SES. After choosing either; Amazon SNS, Amazon CloudWatch, or Amazon Kinesis Firehose as the AWS service to receive the open and click metrics, you would only need to select a new configuration set to successfully enable these new features for any emails you want to send.

AWS Solutions: Serverless Image Handler & AWS Ops Automator

The AWS Solution Builder team has been hard at work helping to make it easier for you all to find answers to common architectural questions to aid in building and running applications on AWS. You can find these solutions on the AWS Answers page. Two new solutions released earlier this fall on AWS Answers are  Serverless Image Handler and the AWS Ops Automator.
Serverless Image Handler was developed to provide a solution to help customers dynamically process, manipulate, and optimize the handling of images on the AWS Cloud. The solution combines Amazon CloudFront for caching, AWS Lambda to dynamically retrieve images and make image modifications, and Amazon S3 bucket to store images. Additionally, the Serverless Image Handler leverages the open source image-processing suite, Thumbor, for additional image manipulation, processing, and optimization.

AWS Ops Automator solution helps you to automate manual tasks using time-based or event-based triggers to automatically such as snapshot scheduling by providing a framework for automated tasks and includes task audit trails, logging, resource selection, scaling, concurrency handling, task completion handing, and API request retries.

The solution includes the following AWS services:

• AWS CloudFormation: templates to launch the core framework of microservices and solution generated task configurations
• Amazon DynamoDB: a table which stores task configuration data to defines the event triggers, resources, and saves the results of the action and the errors.
• Amazon CloudWatch Logs: provides logging to track warning and error messages
• Amazon SNS: topic to send messages to a subscribed email address to which to send the logging information from the solution

Have fun exploring and coding.

– Tara

As I was preparing to write this post, I took a nostalgic look at the blog post I wrote when we launched AWS Direct Connect back in 2012. We created Direct Connect after our enterprise customers asked us to allow them to establish dedicated connections to an AWS Region in pursuit of enhanced privacy, additional data transfer bandwidth, and more predictable data transfer performance. Starting from one AWS Region and a single colo, Direct Connect is now available in every public AWS Region and accessible from dozens of colos scattered across the world (over 60 locations at last count). Our customers have taken to Direct Connect wholeheartedly and we have added features such as Link Aggregation, Amazon EFS support, CloudWatch monitoring, and HIPAA eligibility. In the past five weeks alone we have added Direct Connect locations in Houston (Texas), Vancouver (Canada), Manchester (UK), Canberra (Australia), and Perth (Australia).

Today we are making Direct Connect simpler and more powerful with the addition of the Direct Connect Gateway. We are also giving Direct Connect customers in any Region the ability to create public virtual interfaces that receive our global IP routes and enable access to the public endpoints for our services and updating the Direct Connect pricing model.

Let’s take a look at each one!

New Direct Connect Gateway
You can use the new Direct Connect Gateway to establish connectivity that spans Virtual Private Clouds (VPCs) spread across multiple AWS Regions. You no longer need to establish multiple BGP sessions for each VPC; this reduces your administrative workload as well as the load on your network devices.

This feature also allows you to connect to any of the participating VPCs from any Direct Connect location, further reducing your costs for making using AWS services on a cross-region basis.

Here is a diagram that illustrates the simplification that you can achieve with a Direct Connect Gateway (each “lock” icon represents a Virtual Private Gateway). Start with this:

And end up like this:

The VPCs that reference a particular Direct Connect Gateway must have IP address ranges that do not overlap. Today, the VPCs must all be in the same AWS account; we plan to make this more flexible in the future.

Each Gateway is a global object that exists across all of the public AWS Regions. All communication between the Regions via the Gateways takes place across the AWS network backbone.

Creating a Direct Connect Gateway
You can create a Direct Connect Gateway from the Direct Connect Console or by calling the CreateDirectConnectGateway function from your code. I’ll use the Console!

I open the Direct Connect Console and click on Direct Connect Gateways to get started:

The list is empty since I don’t have any Gateways yet. Click on Create Direct Connect Gateway to change that:

I give my Gateway a name, enter a private ASN for my network, then click on Create. The ASN (Autonomous System Number) must be in one of the ranges defined as private in RFC 6996:

My new Gateway will appear in the other AWS Regions within a moment or two:

I have a Direct Connect Connection in Ohio that I will use to create my VIF:

Now I create a private VIF that references the Gateway and the Connection:

It is ready to use within seconds:

I already have a pair of VPCs with non-overlapping CIDRs, and a Virtual Private Gateway attached to each one. Here are the VPCs (since this is a demo I’ll show both in the same Region for convenience):

And the Virtual Private Gateways:

I return to the Direct Connect Console and navigate to the Direct Connect Gateways. I select my Gateway and choose Associate Virtual Private Gateway from the Actions menu:

Then I select both of my Virtual Private Gateways and click on Associate:

If, as would usually be the case, my VPCs are in distinct AWS Regions, the same procedure would apply. For this blog post it was easier to show you the operations once rather than twice.

The Virtual Gateway association is complete within a minute or so (the state starts out as associating):

When the state transitions to associated, traffic can flow between your on-premises network and your VPCs, over your AWS Direct Connect connection, regardless of the AWS Regions where your VPCs reside.

Public Virtual Interfaces for Service Endpoints
You can now create Public Virtual Interfaces that will allow you to access AWS public service endpoints for AWS services running in any AWS Region (except AWS China Region) over Direct Connect. These interfaces receive (via BGP) Amazon’s global IP routes. You can create these interfaces in the Direct Connect Console; start by selecting the Public option:

Updated Pricing Model
In light of the ever-expanding number of AWS Regions and AWS Direct Connect locations, data transfer pricing is now based on the location of the Direct Connect and the source AWS Region. The new pricing is simpler that the older model which was based on AWS Direct Connect locations.

Now Available
This new feature is available today and you can start to use it right now. You can create and use Direct Connect Gateways at no charge; you pay the usual Direct Connect prices for port hours and data transfer.

— Jeff;

Nine years ago I showed you how you could Distribute Your Content with Amazon CloudFront. We launched CloudFront in 2008 with 14 Points of Presence and have been expanding rapidly ever since. Today I am pleased to announce the opening of our 100^th Point of Presence, the fifth one in Tokyo and the sixth in Japan. With 89 Edge Locations and 11 Regional Edge Caches, CloudFront now supports traffic generated by millions of viewers around the world.

23 Countries, 50 Cities, and Growing
Those 100 Points of Presence span the globe, with sites in 50 cities and 23 countries. In the past 12 months we have expanded the size of our network by about 58%, adding 37 Points of Presence, including nine in the following new cities:

• Berlin, Germany
• Minneapolis, Minnesota, USA
• Prague, Czech Republic
• Boston, Massachusetts, USA
• Munich, Germany
• Vienna, Austria
• Kuala Lumpur, Malaysia
• Philadelphia, Pennsylvania, USA
• Zurich, Switzerland

We have even more in the works, including an Edge Location in the United Arab Emirates, currently planned for the first quarter of 2018.

Innovating for Our Customers
As I mentioned earlier, our network consists of a mix of Edge Locations and Regional Edge Caches. First announced at re:Invent 2016, the Regional Edge Caches sit between our Edge Locations and your origin servers, have even more memory than the Edge Locations, and allow us to store content close to the viewers for rapid delivery, all while reducing the load on the origin servers.

While locations are important, they are just a starting point. We continue to focus on security with the recent launch of our Security Policies feature and our announcement that CloudFront is a HIPAA-eligible service. We gave you more content-serving and content-generation options with the launch of Lambda@Edge, letting you run AWS Lambda functions close to your users.

We have also been working to accelerate the processing of cache invalidations and configuration changes. We now accept invalidations within milliseconds of the request and confirm that the request has been processed world-wide, typically within 60 seconds. This helps to ensure that your customers have access to fresh, timely content!

Visit our Getting Started with Amazon CloudFront page for sign-up information, tutorials, webinars, on-demand videos, office hours, and more.

— Jeff;

In 2015, the Centers for Medicare and Medicaid Services (CMS) reported that healthcare spending made up 17.8% of the U.S. GDP – that’s almost $3.2 trillion or $9,990 per person. By 2025, the CMS estimates this number will increase to nearly 20%. As cloud technology evolves in the healthcare and life science industries, we are seeing how companies of all sizes are using AWS to provide powerful and innovative solutions to customers across the globe. This month we are excited to feature the following startups:

• ClearCare – helping home care agencies operate efficiently and grow their business.
• DNAnexus – providing a cloud-based global network for sharing and managing genomic data.

ClearCare (San Francisco, CA)

ClearCare envisions a future where home care is the only choice for aging in place. Home care agencies play a critical role in the economy and their communities by significantly lowering the overall cost of care, reducing the number of hospital admissions, and bending the cost curve of aging. Patients receiving home care typically have multiple chronic conditions and functional limitations, driving over $190 billion in healthcare spending in the U.S. each year. To offset these costs, health insurance payers are developing in-home care management programs for patients. ClearCare’s goal is to help home care agencies leverage technology to improve costs, outcomes, and quality of life for the aging population. The company’s powerful software platform is specifically designed for use by non-medical, in-home care agencies to manage their businesses.

Founder and CEO Geoff Nudd created ClearCare because of his own grandmother’s need for care. Keeping family members and caregivers up to date on a loved one’s well being can be difficult, so Geoff created what is now ClearCare’s Family Room, which enables caregivers and agency staff to check schedules and receive real-time updates about what’s happening in the home. Since then, agencies have provided feedback on others areas of their businesses that could be streamlined. ClearCare has now built over 20 modules to help home care agencies optimize operations with services including a telephony service, billing and payroll, and more. ClearCare now serves over 4,000 home care agencies, representing 500,000 caregivers and 400,000 seniors.

Using AWS, ClearCare is able to spin up reliable infrastructure for proofs of concept and iterate on those systems to quickly get value to market. The company runs many AWS services including Amazon Elasticsearch Service, Amazon RDS, and Amazon CloudFront. Amazon EMR and Amazon Athena have enabled ClearCare to build a Hadoop-based ETL and data warehousing system that processes terabytes of data each day. By utilizing these managed services, ClearCare has been able to go from concept to customer delivery in less than three months.

To learn more about ClearCare, check out their website.

DNAnexus (Mountain View, CA)

DNAnexus is accelerating the application of genomic data in precision medicine by providing a cloud-based platform for sharing and managing genomic and biomedical data and analysis tools. The company was founded in 2009 by Stanford graduate student Andreas Sundquist and two Stanford professors Arend Sidow and Serafim Batzoglou, to address the need for scaling secondary analysis of next-generation sequencing (NGS) data in the cloud. The founders quickly learned that users needed a flexible solution to build complex analysis workflows and tools that enable them to share and manage large volumes of data. DNAnexus is optimized to address the challenges of security, scalability, and collaboration for organizations that are pursuing genomic-based approaches to health, both in clinics and research labs. DNAnexus has a global customer base – spanning North America, Europe, Asia-Pacific, South America, and Africa – that runs a million jobs each month and is doubling their storage year-over-year. The company currently stores more than 10 petabytes of biomedical and genomic data. That is equivalent to approximately 100,000 genomes, or in simpler terms, over 50 billion Facebook photos!

DNAnexus is working with its customers to help expand their translational informatics research, which includes expanding into clinical trial genomic services. This will help companies developing different medicines to better stratify clinical trial populations and develop companion tests that enable the right patient to get the right medicine. In collaboration with Janssen Human Microbiome Institute, DNAnexus is also launching Mosaic – a community platform for microbiome research.

AWS provides DNAnexus and its customers the flexibility to grow and scale research programs. Building the technology infrastructure required to manage these projects in-house is expensive and time-consuming. DNAnexus removes that barrier for labs of any size by using AWS scalable cloud resources. The company deploys its customers’ genomic pipelines on Amazon EC2, using Amazon S3 for high-performance, high-durability storage, and Amazon Glacier for low-cost data archiving. DNAnexus is also an AWS Life Sciences Competency Partner.

Learn more about DNAnexus here.

—

-Tina

Our Health Customer Stories page lists just a few of the many customers that are building and running healthcare and life sciences applications that run on AWS. Customers like Verge Health, Care Cloud, and Orion Health trust AWS with Protected Health Information (PHI) and Personally Identifying Information (PII) as part of their efforts to comply with HIPAA and HITECH.

Sixteen More Services
In my last HIPAA Eligibility Update I shared the news that we added eight additional services to our list of HIPAA eligible services. Today I am happy to let you know that we have added another sixteen services to the list, bringing the total up to 46. Here are the newest additions, along with some short descriptions and links to some of my blog posts to jog your memory:

Amazon Aurora with PostgreSQL Compatibility – This brand-new addition to Amazon Aurora allows you to encrypt your relational databases using keys that you create and manage through AWS Key Management Service (KMS). When you enable encryption for an Amazon Aurora database, the underlying storage is encrypted, as are automated backups, read replicas, and snapshots. Read New – Encryption at Rest for Amazon Aurora to learn more.

Amazon CloudWatch Logs – You can use the logs to monitor and troubleshoot your systems and applications. You can monitor your existing system, application, and custom log files in near real-time, watching for specific phrases, values, or patterns. Log data can be stored durably and at low cost, for as long as needed. To learn more, read Store and Monitor OS & Application Log Files with Amazon CloudWatch and Improvements to CloudWatch Logs and Dashboards.

Amazon Connect – This self-service, cloud-based contact center makes it easy for you to deliver better customer service at a lower cost. You can use the visual designer to set up your contact flows, manage agents, and track performance, all without specialized skills. Read Amazon Connect – Customer Contact Center in the Cloud and New – Amazon Connect and Amazon Lex Integration to learn more.

Amazon ElastiCache for Redis – This service lets you deploy, operate, and scale an in-memory data store or cache that you can use to improve the performance of your applications. Each ElastiCache for Redis cluster publishes key performance metrics to Amazon CloudWatch. To learn more, read Caching in the Cloud with Amazon ElastiCache and Amazon ElastiCache – Now With a Dash of Redis.

Amazon Kinesis Streams – This service allows you to build applications that process or analyze streaming data such as website clickstreams, financial transactions, social media feeds, and location-tracking events. To learn more, read Amazon Kinesis – Real-Time Processing of Streaming Big Data and New: Server-Side Encryption for Amazon Kinesis Streams.

Amazon RDS for MariaDB – This service lets you set up scalable, managed MariaDB instances in minutes, and offers high performance, high availability, and a simplified security model that makes it easy for you to encrypt data at rest and in transit. Read Amazon RDS Update – MariaDB is Now Available to learn more.

Amazon RDS SQL Server – This service lets you set up scalable, managed Microsoft SQL Server instances in minutes, and also offers high performance, high availability, and a simplified security model. To learn more, read Amazon RDS for SQL Server and .NET support for AWS Elastic Beanstalk and Amazon RDS for Microsoft SQL Server – Transparent Data Encryption (TDE) to learn more.

Amazon Route 53 – This is a highly available Domain Name Server. It translates names like www.example.com into IP addresses. To learn more, read Moving Ahead with Amazon Route 53.

AWS Batch – This service lets you run large-scale batch computing jobs on AWS. You don’t need to install or maintain specialized batch software or build your own server clusters. Read AWS Batch – Run Batch Computing Jobs on AWS to learn more.

AWS CloudHSM – A cloud-based Hardware Security Module (HSM) for key storage and management at cloud scale. Designed for sensitive workloads, CloudHSM lets you manage your own keys using FIPS 140-2 Level 3 validated HSMs. To learn more, read AWS CloudHSM – Secure Key Storage and Cryptographic Operations and AWS CloudHSM Update – Cost Effective Hardware Key Management at Cloud Scale for Sensitive & Regulated Workloads.

AWS Key Management Service – This service makes it easy for you to create and control the encryption keys used to encrypt your data. It uses HSMs to protect your keys, and is integrated with AWS CloudTrail in order to provide you with a log of all key usage. Read New AWS Key Management Service (KMS) to learn more.

AWS Lambda – This service lets you run event-driven application or backend code without thinking about or managing servers. To learn more, read AWS Lambda – Run Code in the Cloud, AWS Lambda – A Look Back at 2016, and AWS Lambda – In Full Production with New Features for Mobile Devs.

Lambda@Edge – You can use this new feature of AWS Lambda to run Node.js functions across the global network of AWS locations without having to provision or manager servers, in order to deliver rich, personalized content to your users with low latency. Read Lambda@Edge – Intelligent Processing of HTTP Requests at the Edge to learn more.

AWS Snowball Edge – This is a data transfer device with 100 terabytes of on-board storage as well as compute capabilities. You can use it to move large amounts of data into or out of AWS, as a temporary storage tier, or to support workloads in remote or offline locations. To learn more, read AWS Snowball Edge – More Storage, Local Endpoints, Lambda Functions.

AWS Snowmobile – This is an exabyte-scale data transfer service. Pulled by a semi-trailer truck, each Snowmobile packs 100 petabytes of storage into a ruggedized 45-foot long shipping container. Read AWS Snowmobile – Move Exabytes of Data to the Cloud in Weeks to learn more (and to see some of my finest LEGO work).

AWS Storage Gateway – This hybrid storage service lets your on-premises applications use AWS cloud storage (Amazon Simple Storage Service (S3), Amazon Glacier, and Amazon Elastic File System) in a simple and seamless way, with storage for volumes, files, and virtual tapes. To learn more, read The AWS Storage Gateway – Integrate Your Existing On-Premises Applications with AWS Cloud Storage and File Interface to AWS Storage Gateway.

And there you go! Check out my earlier post for a list of resources that will help you to build applications that comply with HIPAA and HITECH.

— Jeff;