AWS Official Blog

Amazon Relational Database Service (RDS) makes it easy for you to set up, run, scale, and maintain a relational database. As is often the case with the high-level AWS model, we take care of all of the details in order to give you the time to focus on your application and your business.

Enhanced Monitoring
Advanced RDS users have asked us for more insight into the inner workings of the service and we are happy to oblige with a new Enhanced Monitoring feature!

After you enable this feature for a database instance, you get access to over 50 new CPU, memory, file system, and disk I/O metrics. You can enable these features on a per-instance basis, and you can choose the granularity (all the way down to 1 second). Here is the list of available metrics:

And here are some of the metrics for one of my database instances:

You can enable this feature for an existing database instance by selecting the instance in the RDS Console and then choosing Modify from the Instance Options menu:

Turn the feature on, pick an IAM role, select the desired granularity, check Apply Immediately, and then click on Continue.

The Enhanced Metrics are ingested into CloudWatch Logs and can be published to Amazon CloudWatch. To do this you will need to set up a metrics extraction filter; read about Monitoring Logs to learn more. Once the metrics are stored in CloudWatch Logs, they can also be processed by third-party analysis and monitoring tools.

Available Now
The new Enhanced Metrics feature is available today in the US East (Northern Virginia), US West (Northern California), US West (Oregon), Europe (Ireland), Europe (Frankfurt), Asia Pacific (Singapore), Asia Pacific (Sydney), and Asia Pacific (Tokyo) regions. It works for MySQL 5.6, MariaDB, and Amazon Aurora, on all instance types except t1.micro and m1.small.

You will pay the usual ingestion and data transfer charges for CloudWatch Logs (see the CloudWatch Logs Pricing page for more info).

We announced AWS Config Rules (Dynamic Compliance Checking for Cloud Resources) at AWS re:Invent and made a preview available to interested customers.

As I noted at the time, you can use these rules to verify that existing and newly launched AWS resources conform to your organization’s security guidelines and best practices without having to spend time manually inspecting them. Instead, you define rules (AWS Lambda functions) that are run when resources are created or changed. The rule has access to the Configuration Item associated with the resource, and can also make calls to other AWS API functions as needed.

Available Now
Today we are making Config Rules accessible to all AWS customers, with initial availability in the US East (Northern Virginia) region and the ability to create up to 25 Config Rules per account.

We added some important features during the preview:

• Support for additional IAM and EC2 resource types. You can now write rules that process IAM Users, Groups, Roles, including customer-managed policies. You can also write rules that process EC2 Dedicated Hosts.
• Lambda Blueprints to help create custom rules. You now have access to blueprints for periodic rules and for rules that run in response to changes.
• CloudFormation Support for Config and for Config Rules. You can now automate the setup of AWS Config for new accounts, and you can automate the creation and configuration of the Config Rules.

Several AWS partners are already making great use of Config Rules in production. For example, Alert Logic, CloudHealth Technologies and Trend Micro Deep Security are using Config Rules as integral parts of their respective flagship products.

Creating a Custom Rule
I will create a rule that inspects an IAM Policy named DBSuperUserPolicy. I want to make sure that the policy has only one user, DBSuperUser, attached to it. If this is the case, the Policy is compliant; otherwise, it is not.

I start by creating a Lambda function using one of the new blueprints. I open up the Lambda Console, click on Create a Lambda function, and locate the blueprint that I want to use. I enter “config” in the search box:

I want my rule to run every time the IAM Policy is changed, so I select the first blueprint, config-rule-change-triggered. Then I enter a name and description for the function:

The code provided in the blueprint includes a function named evaluateCompliance. This function is the heart of the rule; it is activated on each configuration change and always returns one of the following strings:

• NOT_APPLICABLE -The rule does not apply to the resource or the resource type that it was given.
• COMPLIANT – The rule is relevant to the resource, and the resource is configured in a compliant way.
• NON_COMPLIANT – The rule is relevant to the resource, and the resource is not configured in a compliant way.

I replace the default implementation of the function with my own. It looks like this:

function evaluateCompliance(configurationItem, ruleParameters, context)
{
  if ((configurationItem.resourceType !== 'AWS::IAM::Policy') || 
      (configurationItem.resourceName !== 'DBSuperUserUsagePolicy'))
    return 'NOT_APPLICABLE';
  if (configurationItem.relationships[0].resourceName === ruleParameters.attachedIAMUser)
  {
    if (configurationItem.relationships[1])
      return 'NON_COMPLIANT';
    else
      return 'COMPLIANT';
  }
  else
    return 'NON_COMPLIANT';
}

This rule is simple and powerful! Let’s go through it, line-by-line.

• Lines 3 and 4 check to see if the rule was invoked on an IAM Policy named DBSuperUserPolicy; line 5 returns NOT_APPLICABLE if this is not the case.
• Line 6 verifies that there is an IAM User attached to the Policy; line 14 returns NON_COMPLIANT if this is not the case.
• Line 8 verifies that there is no more than one User attached to the Policy; line 9 returns NON_COMPLIANT if this is not the case.
• Line 11 returns COMPLIANT to indicate that the Policy has exactly one IAM User attached, and is therefore compliant.

As you can see, this simple, 15-line function is all that you need to enforce an organization-wide policy across all of your AWS resources.

I also need to create an IAM Role so that the function can send the results to AWS Config:

If the function needs to access other AWS resources or APIs, I would amend the policy document accordingly. I select the role, confirm that I want to create the function, and I am already halfway there:

With the function in place, I need to arrange for Config Rules to call it as needed. I visit the Config Rules Console and click on Add Rule:

At this point I can choose one of the seven AWS managed rules, or I can click on Add custom rule (sounds good to me):

Now I name my rule, point it at my Lambda function (via its ARN), and indicate that I want it to be run when the configuration of an IAM Policy changes:

The rule will be evaluated within a few minutes; I can verify that my function has been invoked by taking a peek at the function’s Monitoring tab in the Lambda console:

From looking at these metrics I can see that the function has been run 7 times (I have that number of customer-managed IAM Policies), that each invocation lasts for less than 1 second, and that the invocations are not raising any errors. So far, so good!

Exercising the Rule
With the function running and attached to the rule, the next step is to make sure that it performs as expected. I create an IAM Policy named DBSuperUserUsagePolicy and attach user jeff to it (this is not in compliance with my rule, and it should be marked as such):

After allowing some time for the rule to be run, I return to the Config Rules console and I can see that there’s an issue:

I can learn more with a click:

If I don’t recall making the change or don’t know who did it (I did, but I was tired and clearly made a mistake), I can click on the Config timeline to learn more:

After some investigation I realize that I was supposed to use DBSuperUser, and update my Policy accordingly:

I wait a few minutes and check the rule details again. I am good to go – my resource is now compliant with my policy:

My change is reflected in the timeline (as you can see, I actually do spend some time away from the keyboard; I created the policy last night and edited it this morning):

Finally, I can see that my Lambda function was invoked after I updated my policy:

CloudFormation Support
You can now use CloudFormation templates to create your Config Rules and your Lambda functions. Here’s how you would create a rule that references a function called VolumeAutoEnableIOComplianceCheck:

"ConfigRuleForVolumeAutoEnableIO": {
      "Type": "AWS::Config::ConfigRule",
      "Properties": {
        "ConfigRuleName": "ConfigRuleForVolumeAutoEnableIO",
        "Scope": {
          "ComplianceResourceId": {"Ref": "Ec2Volume"},
          "ComplianceResourceTypes": ["AWS::EC2::Volume"]
        },
        "Source": {
          "Owner": "CUSTOM_LAMBDA",
          "SourceDetails": [{
              "EventSource": "aws.config",
              "MessageType": "ConfigurationItemChangeNotification"
          }],
          "SourceIdentifier": {"Fn::GetAtt": ["VolumeAutoEnableIOComplianceCheck", "Arn"]}
        }
      },
      "DependsOn": "ConfigPermissionToCallLambda"
    },

Partner Support
As I mentioned earlier, several AWS partners are already making great use of this feature. Here’s some more information on their offerings:

Alert Logic Cloud Insight allows customers to use AWS Config to configure checks for additional custom vulnerabilities. Learn more on the Alert Logic Partner Page.

CloudHealth Technologies stores AWS infrastructure configuration history and supports searching of changes by group, access to historical changes, and a history of asset configuration. Learn more on the CloudHealth Technologies Partner Page.

Trend Micro provides a comprehensive set of security controls. Learn more on the Trend Micro Partner Page.

Availability and Pricing
Config Rules is now available in the US East (Northern Virginia) region and you can start using it today. Now that it is generally available, you will be charged for usage as described on the Config Rules Pricing page.

A few months ago, I wrote about AWS IoT (see AWS IoT – Cloud Services for Connected Devices) and talked about how we are working to make sure that AWS is well-equipped to support many different types of IoT devices and applications. At that time we launched AWS IoT in beta form and invited interested developers to sign up and to start getting experience with the service.

We built AWS IoT because connected devices are proliferating. They are in your house, your car, your office, your school, and perhaps even in your body! Like some of our more advanced customers, we have been building systems around connected devices for quite some time. Our experience with Amazon Robotics, drones (Amazon Prime Air), the Amazon Echo, the Dash Button, and multiple generations of Kindles has given us a well-informed perspective on how to serve this really important emerging market. Behind the scenes, AWS services such as AWS Lambda, Amazon API Gateway, Amazon DynamoDB, Amazon Kinesis, Amazon Simple Storage Service (S3), and Amazon Redshift provide the responsive, highly scalable infrastructure needed to build a robust IoT application.

When we talked to our customers and to our own engineers, we learned quite a bit about the pain points that add complexity and development time to IoT applications. They told us that connecting devices to the cloud is overly complex due to the variety of SDKs and protocols that they need to support in a secure and scalable fashion. Making this even more difficult is the fact that many devices “feature” intermittent connectivity to the Internet, even as application logic shifts from the device to the cloud. Finally, the sheer volume of data generated by the sensors attached to the devices mandates a Big Data approach to storage, analytics, and visualization.

These are, to be sure, some steep requirements. As you can read in my post above, we have designed AWS IoT with all of them in mind.

Now Available
I am happy to be able to announce that the beta period is over and that AWS IoT is now generally available. Many AWS customers are already building apps and creating new businesses around IoT. Here are a couple of examples:

• The Philips HealthSuite digital platform collects, analyzes, and stores 15 petabytes of patient data (case study).
• Scout Alarm uses AWS IoT to support an ever-growing set of self-installed, wireless home security systems, allowing them to focus on the user experience instead of on the infrastructure.

During the beta, we added a pair of important features to AWS IoT:

• AWS Mobile SDK for iOS now supports AWS IoT (Android support is in the works).
• IPv6 support for exchanging messages between devices and the AWS IoT Device Gateway.

IoT Use Cases
Earlier this week I sat down with a couple of members of the IoT team. They told me that our customers are planning to use AWS IoT to support many industries and use cases! Here’s a sampling:

• Agriculture
• Cars & trucks
• Consumer devices
• Gaming
• Home automation
• Logistics
• Medical
• Municipal infrastructure
• Oil & gas
• Robotics

Get Started Today
To learn more about AWS IoT and how you can put it to use in your environment, hop on over to the Getting Started page. You may also want to read the AWS IoT FAQs and study the AWS IoT Documentation.

Many of our customers have been asking us for data and tools to allow them to better understand and manage their AWS costs.

New Reports
Today we are introducing a set of new AWS Cost and Usage Reports that provide you with comprehensive data about products, pricing, and usage. The reports allow you to understand individual costs and to analyze them in greater detail. For example, you can view your EC2 costs by instance type and then drill-down in order to understand usage by operating system, instance type, and purchase option (On-Demand, Reserved, or Spot).

The new reports are generated in CSV form and can be customized. You can select the data included in each report, decide whether you want it aggregated across an hour or a day, and then request delivery to one of your S3 buckets, with your choice of ZIP or GZIP compression. The data format is normalized so that each discrete cost component is presented in an exclusive column.

You can easily upload the reports to Amazon Redshift and then run queries against the data using business intelligence and data visualization tools including Amazon QuickSight.

Creating a Report
To create a report, head on over to the AWS Management Console, and choose Billing & Cost Management from the menu in the top-right:

Then click on Reports in the left navigation:

Click on Create report to create your first report:

Enter a name for your report, pick a time unit, and decide whether you want to include Resource IDs (more detail and a bigger file) or not:

Now choose your delivery options: pick an S3 bucket (you’ll need to set the permissions per the sample policy), set a prefix if you’d like, and select the desired compression (GZIP or ZIP):

Click on Next, review your choices, and then create your report. It will become visible on the AWS Cost and Usage Reports page:

A fresh report will be delivered to the bucket within 24 hours. Additional reports will be provided every 24 hours (or less) thereafter.

From there you can transfer them to Redshift using a AWS Data Pipeline job or some code triggered by a AWS Lambda function, and then analyze them using the BI or data visualization tool of your choice.

Visualizing the Data
Here are some sample visualizations, courtesy of Amazon QuickSight. Looking at our EC2 spend by instance type gives an overall picture of our spending:

Viewing it over time shows that spending varies considerably from day to day:

Learn More
To learn more, read about Understanding Your Usage with Billing Reports.

My colleague Sivakanth Mundru wrote the guest post below in order to share news of some important new features for AWS CloudTrail.

As many of you know AWS CloudTrail provides visibility into API activity in your AWS account and enables you to answer important questions such as which user made an API call or which resources were acted upon in an API call. Today, we are happy to deliver two features that are many of you asked for:

1. The ability to turn on CloudTrail across all AWS regions.
2. Support for multiple trails.

Turn on CloudTrail in All Regions
Until now, you had to turn on CloudTrail for each desired region. Many of you provided feedback to us that this is time consuming, and asked for the ability to turn on CloudTrail in all regions with few clicks.

Starting immediately, you can simply specify that a trail will apply to all regions and CloudTrail will automatically create the same trail in each region, record and process log files in each region, and deliver log files from all regions to the S3 bucket or (optionally) the CloudWatch Logs log group you specified.

To be a bit more specific, “all” refers to the regions within a single AWS partition. The US East (Northern Virginia), US West (Northern California), US West (Oregon), Europe (Ireland), Europe (Frankfurt), Asia Pacific (Sydney), Asia Pacific (Sydney), Asia Pacific (Tokyo), and South America (Brazil) regions are all in the aws partition; the Beijing (China) region is in the aws-cn partition (read Amazon Resource Names (ARNs) and AWS Service Namespaces to learn more). The features described in this post apply to the aws partition.

Future Proof for New Regions
In addition to turning on CloudTrail for all existing regions, when AWS launches a new region  CloudTrail will create the trail in the new region and turn it on. As a result, you will receive log files containing API activity for your AWS account in the new region without taking any action.

Here’s how you turn on CloudTrail in all regions via the AWS Management Console:

Support for Multiple Trails
CloudTrail log files enable you to troubleshoot operational or security issues in your AWS account and help you demonstrate compliance with your internal policies or external standards. Different stakeholders have different needs. With support for multiple trails, different stakeholders in the company can create and manage their own trails for their own needs. For example:

• A security administrator can create a trail that applies to all regions and encrypt the log files with one KMS key.
• A developer can create a trail that applies to one region, for example Asia Pacific (Sydney), and configure CloudWatch alarms to receive notifications of specific API activity.
• An IT auditor can create a trail that applies to one region, say Europe (Frankfurt), and configure log file integrity validation to positively assert that log files are not changed since CloudTrail delivered the log files to an S3 bucket.

Here’s what this would look like:

You can create up to 5 trails per region (a trail that applies to all regions exists in each region and counted as 1 trail per region).

As part of today’s launch we are announcing support for resource level permissions so that you can prescribe granular access control policies on which users can or cannot take particular actions on a given trail. For more details and sample policies, see the CloudTrail documentation.

Viewing and Managing Trails Across Regions
We are also announcing an important enhancement to the CloudTrail Console!

You can now view and manage trails across all regions in a partition, no matter which region you are in. You will see all the trails for your account in every region.  You can click on the trail name and CloudTrail will navigate to the trail configuration page automatically:

As you can see, the trail named Allregionstrail applies to all regions. This means that the Allregionstrail exists in every region and log files for all regions are recorded and delivered to one S3 bucket and an optional CloudWatch Logs log group. Other trails are specific to a region and log files for those specific regions are recorded and delivered as per the trail configuration. You can click on a trail name to view, edit or delete a trail.

Pricing
All new and existing AWS customers can create one trail per region and record API activity for services supported by CloudTrail as a part of the free tier. The free tier does not have an expiration.

A trail that applies to all regions exists in each region and counted as 1 trail per region.

You pay $2.00 per 100,000 events recorded in each additional trail. There is no charge for creating additional trails.

— Sivakanth Mundru, Senior Product Manager

Amazon CloudFront helps you to get your content to your users at high speed with low latency.

Today we are making CloudFront even better with the addition of support for Gzip compression. After you enable it for a particular CloudFront distribution, text and binary content will be compressed at the edge and returned in response to requests that indicate that compressed content is preferred (most modern browsers do this automatically).

Your pages will load more quickly, content will download faster, and your CloudFront data transfer charges may be reduced as well. For a typical web page composed of a mix of text, scripts, and images, the overall payload reduction can approach 80%.

I tested this new feature on this very blog! Here is the data transfer without compression:

And here it is with compression:

As you can see from the browser’s status bar, Gzip compression reduced total download size from 792 KB to 177 KB (a 77% reduction). Download time was reduced from 846 ms to 446 ms (almost 50%).

Enabling Gzip Compression
You can enable this feature in a minute! Simply open up the CloudFront Console, locate your distribution, and set Compress Objects Automatically to Yes in the Behavior options:

To learn more, read about Serving Compressed Files.

Available Now
This feature is available now and you can start using it today! There is no extra charge for the compression; your CloudFront data transfer charges may actually go down (the specifics depend on the proportion of compressed to uncompressed requests, of course).

AWS Marketplace is an online store that helps you to find, buy, and immediately start using a very wide variety of applications on AWS (some of the more popular categories are Network Infrastructure, Security, and Big Data).

Up until now, running an application from AWS Marketplace was essentially equivalent to launching a single, self-contained Amazon EC2 instance. This was a good starting point, but it was not sufficient to deal with more sophisticated applications that run across a cluster of instances and/or require additional AWS resources such as Auto Scaling groups, Elastic Load Balancers, SQL database instances, an advanced network configuration, message queues, and so forth.

Support for Clusters and AWS Resources
To address this customer need, an application in the AWS Marketplace can now be represented by up to three AWS CloudFormation templates, each created by the application vendor, and each with a distinct set of deployment options. Before you actually launch a template-backed product, you will see a list of the AWS resources that will be created, along with an estimate of the monthly costs. Vendors also have the option to provide the traditional AMI-powered option alongside the new and more powerful template-powered options.

Initial Application Support
We have been working with a group of application vendors to make their applications available in this new and more flexible fashion. Here’s what you can launch today:

• Sophos –Sophos UTM 9 Autoscaling.
• Tibco – Jaspersoft for AWS with Multi-Tenancy (Hourly).
• NetApp – OnCommand Cloud Manager on Linux.
• MapR – Enterprise Database Edition Plus.

We’ll be adding more applications in the very near future.

As you can see from this screen shot, the deployment options are clearly visible, as is the estimated cost:

Template Power!
Because this new option makes use of CloudFormation, sophisticated users have access to some interesting new features. The launch process makes use of the CloudFormation console and supports the usual prompting for parameters and generation of multiple output values. Templates can be downloaded, inspected, and even edited in the CloudFormation Designer that we launched earlier this year.

Note to Application Vendors
If you are already selling your products on the Marketplace and would like to take advantage of this new option, contact your AWS BDM (Business Development Manager) or email the AWS Marketplace team at aws-marketplace-seller-ops@amazon.com. If you are new to AWS Marketplace, start by reading our Sell on Marketplace page.

You can use Amazon Virtual Private Cloud to create a logically isolated section of the AWS Cloud. Within the VPC, you can define your desired IP address range, create subnets, configure route tables, and so forth. You can also use a virtual private gateway to connect the VPC to your existing on-premises network using a hardware Virtual Private Network (VPN) connection.

An interesting network challenge arises when EC2 instances in a private VPC subnet need to connect to the Internet. Because the subnet is private, the IP addresses assigned to the instances cannot be used in public. Instead, it is necessary to use Network Address Translation (NAT) to map the private IP addresses to a public address on the way out, and then map the public IP address to the private address on the return trip.

New Managed NAT Gateway
Performing this translation at scale can be challenging. In order to simplify the task (and, as usual, to let you spend more time on your application and on your business), we are launching a new Managed NAT Gateway for AWS!

Instead of configuring, running, monitoring, and scaling a cluster of EC2 instances (you’d need at least 2 in order to ensure high availability), you can now create and configure a gateway with a couple of clicks.

The gateway has built-in redundancy for high availability. Each gateway that you create can handle up to 10 Gbps of bursty TCP, UDP, and ICMP traffic, and is managed by Amazon. You control the public IP address by assigning an Elastic IP Address when you create the gateway.

Creating a Managed NAT Gateway
Let’s create a Managed NAT Gateway! Open up the VPC Console, and take a peek at the navigation area on the left. Locate and click on NAT Gateways:

Then click on Create NAT Gateway and choose one of your subnets:

Choose one of your existing Elastic IP addresses, or create a new one:

Then click on Create a NAT Gateway, and observe the confirmation:

As you can see from the confirmation, you will need to edit your VPC’s route tables to send traffic destined for the Internet toward the gateway. The gateway’s internal (private) IP address will be chosen automatically, and will be on the subnet associated with the gateway. Here’s a sample route table:

And that’s all you need to do. You don’t need to size, scale, or manage the gateway.

You can use VPC Flow Logs to capture the traffic flowing through your gateway, and then use the information in the logs to create CloudWatch metrics based on packets, bytes, and protocols. You can use the following filter pattern as a starting point (be sure to enter actual values for ENI_ID and NGW_IP):

[version, account_id, interface_id=ENI_ID, src_addr, dst_addr=NGW_IP, src_port, dst_port, protocol, packets, bytes, start, end, action, log_status]

The resulting graph will look like this:

If you create a new VPC using the VPC Wizard, it will offer to create a NAT Gateway and the route table rules for you. This makes the setup process even easier!

To learn more, read about the VPC NAT Gateway in the VPC User Guide.

Pricing and Availability
You can start using this new feature today in the US East (Northern Virginia), US West (Oregon), US West (Northern California), Europe (Ireland), Asia Pacific (Singapore), Asia Pacific (Sydney), and Asia Pacific (Tokyo) regions.

Pricing starts at $0.045 per NAT gateway hour plus data processing and data transfer charges. Data processing costs are based on the amount of data processed by the NAT Gateway; data transfer costs are the usual costs to move data between an EC2 instance and the Internet. For more information, read about VPC Pricing.

Back when I was young, InfoWorld was a tabloid-sized journal that chronicled the growth of the PC industry. Every week I would await the newest issue and read it cover to cover, eager to learn all about the latest and greatest hardware and software. I always enjoyed and appreciated the reviews — they were unfailingly deep, objective, and helpful.

With this as background, I am really happy to be able to let you know that the team at InfoWorld recently put Amazon Aurora through its paces, wrote a detailed review, and named it an Editor’s Choice. They succinctly and accurately summarized the architecture, shared customer feedback from AWS re:Invent, and ran an intensive benchmark, concluding that:

This level of performance is far beyond any I’ve seen from other open source SQL databases, and it was achieved at far lower cost than you would pay for an Oracle database of similar power.

We’re very proud of Amazon Aurora and I think you’ll understand why after you read this review.

The AWS Trusted Advisor helps you to provision and configure your AWS resources so as to improve system performance and reliability, increase security, and optimize for cost. We have added some new checks and improved an existing one in order to make Trusted Advisor even more useful to you. Here is a summary of the changes:

The Service Limits check now reports on your usage of EC2 On-Demand instances:

This check is available to all users of Trusted Advisor. The remaining checks are available to customers who are using AWS Support API at the Business or Enterprise level.

The S3 Bucket Logging Configuration check now looks to see if server access logging has been configured for each bucket:

The new EC2 to EBS Throughput check looks for EBS volumes that might be affected by the throughput capacity of the EC2 instances:

The new CloudFront Alternate Domains check looks at the DNS settings for alternate domains on your CloudFront distributions:

The new CloudFront SSL Certificate on the Origin Server check looks for SSL certificates that are expired, about to expire, or that use outdated encryption:

The new IAM Access Key Rotation check looks for IAM keys that have not been rotated in the last 90 days:

The new checks are available now and you can benefit from them today. Visit the AWS Trusted Advisor to learn more.