AWS Official Blog

Many of our customers have been asking us for data and tools to allow them to better understand and manage their AWS costs.

New Reports
Today we are introducing a set of new AWS Cost and Usage Reports that provide you with comprehensive data about products, pricing, and usage. The reports allow you to understand individual costs and to analyze them in greater detail. For example, you can view your EC2 costs by instance type and then drill-down in order to understand usage by operating system, instance type, and purchase option (On-Demand, Reserved, or Spot).

The new reports are generated in CSV form and can be customized. You can select the data included in each report, decide whether you want it aggregated across an hour or a day, and then request delivery to one of your S3 buckets, with your choice of ZIP or GZIP compression. The data format is normalized so that each discrete cost component is presented in an exclusive column.

You can easily upload the reports to Amazon Redshift and then run queries against the data using business intelligence and data visualization tools including Amazon QuickSight.

Creating a Report
To create a report, head on over to the AWS Management Console, and choose Billing & Cost Management from the menu in the top-right:

Then click on Reports in the left navigation:

Click on Create report to create your first report:

Enter a name for your report, pick a time unit, and decide whether you want to include Resource IDs (more detail and a bigger file) or not:

Now choose your delivery options: pick an S3 bucket (you’ll need to set the permissions per the sample policy), set a prefix if you’d like, and select the desired compression (GZIP or ZIP):

Click on Next, review your choices, and then create your report. It will become visible on the AWS Cost and Usage Reports page:

A fresh report will be delivered to the bucket within 24 hours. Additional reports will be provided every 24 hours (or less) thereafter.

From there you can transfer them to Redshift using a AWS Data Pipeline job or some code triggered by a AWS Lambda function, and then analyze them using the BI or data visualization tool of your choice.

Visualizing the Data
Here are some sample visualizations, courtesy of Amazon QuickSight. Looking at our EC2 spend by instance type gives an overall picture of our spending:

Viewing it over time shows that spending varies considerably from day to day:

Learn More
To learn more, read about Understanding Your Usage with Billing Reports.

My colleague Sivakanth Mundru wrote the guest post below in order to share news of some important new features for AWS CloudTrail.

As many of you know AWS CloudTrail provides visibility into API activity in your AWS account and enables you to answer important questions such as which user made an API call or which resources were acted upon in an API call. Today, we are happy to deliver two features that are many of you asked for:

1. The ability to turn on CloudTrail across all AWS regions.
2. Support for multiple trails.

Turn on CloudTrail in All Regions
Until now, you had to turn on CloudTrail for each desired region. Many of you provided feedback to us that this is time consuming, and asked for the ability to turn on CloudTrail in all regions with few clicks.

Starting immediately, you can simply specify that a trail will apply to all regions and CloudTrail will automatically create the same trail in each region, record and process log files in each region, and deliver log files from all regions to the S3 bucket or (optionally) the CloudWatch Logs log group you specified.

To be a bit more specific, “all” refers to the regions within a single AWS partition. The US East (Northern Virginia), US West (Northern California), US West (Oregon), Europe (Ireland), Europe (Frankfurt), Asia Pacific (Sydney), Asia Pacific (Sydney), Asia Pacific (Tokyo), and South America (Brazil) regions are all in the aws partition; the Beijing (China) region is in the aws-cn partition (read Amazon Resource Names (ARNs) and AWS Service Namespaces to learn more). The features described in this post apply to the aws partition.

Future Proof for New Regions
In addition to turning on CloudTrail for all existing regions, when AWS launches a new region  CloudTrail will create the trail in the new region and turn it on. As a result, you will receive log files containing API activity for your AWS account in the new region without taking any action.

Here’s how you turn on CloudTrail in all regions via the AWS Management Console:

Support for Multiple Trails
CloudTrail log files enable you to troubleshoot operational or security issues in your AWS account and help you demonstrate compliance with your internal policies or external standards. Different stakeholders have different needs. With support for multiple trails, different stakeholders in the company can create and manage their own trails for their own needs. For example:

• A security administrator can create a trail that applies to all regions and encrypt the log files with one KMS key.
• A developer can create a trail that applies to one region, for example Asia Pacific (Sydney), and configure CloudWatch alarms to receive notifications of specific API activity.
• An IT auditor can create a trail that applies to one region, say Europe (Frankfurt), and configure log file integrity validation to positively assert that log files are not changed since CloudTrail delivered the log files to an S3 bucket.

Here’s what this would look like:

You can create up to 5 trails per region (a trail that applies to all regions exists in each region and counted as 1 trail per region).

As part of today’s launch we are announcing support for resource level permissions so that you can prescribe granular access control policies on which users can or cannot take particular actions on a given trail. For more details and sample policies, see the CloudTrail documentation.

Viewing and Managing Trails Across Regions
We are also announcing an important enhancement to the CloudTrail Console!

You can now view and manage trails across all regions in a partition, no matter which region you are in. You will see all the trails for your account in every region.  You can click on the trail name and CloudTrail will navigate to the trail configuration page automatically:

As you can see, the trail named Allregionstrail applies to all regions. This means that the Allregionstrail exists in every region and log files for all regions are recorded and delivered to one S3 bucket and an optional CloudWatch Logs log group. Other trails are specific to a region and log files for those specific regions are recorded and delivered as per the trail configuration. You can click on a trail name to view, edit or delete a trail.

Pricing
All new and existing AWS customers can create one trail per region and record API activity for services supported by CloudTrail as a part of the free tier. The free tier does not have an expiration.

A trail that applies to all regions exists in each region and counted as 1 trail per region.

You pay $2.00 per 100,000 events recorded in each additional trail. There is no charge for creating additional trails.

— Sivakanth Mundru, Senior Product Manager

Amazon CloudFront helps you to get your content to your users at high speed with low latency.

Today we are making CloudFront even better with the addition of support for Gzip compression. After you enable it for a particular CloudFront distribution, text and binary content will be compressed at the edge and returned in response to requests that indicate that compressed content is preferred (most modern browsers do this automatically).

Your pages will load more quickly, content will download faster, and your CloudFront data transfer charges may be reduced as well. For a typical web page composed of a mix of text, scripts, and images, the overall payload reduction can approach 80%.

I tested this new feature on this very blog! Here is the data transfer without compression:

And here it is with compression:

As you can see from the browser’s status bar, Gzip compression reduced total download size from 792 KB to 177 KB (a 77% reduction). Download time was reduced from 846 ms to 446 ms (almost 50%).

Enabling Gzip Compression
You can enable this feature in a minute! Simply open up the CloudFront Console, locate your distribution, and set Compress Objects Automatically to Yes in the Behavior options:

To learn more, read about Serving Compressed Files.

Available Now
This feature is available now and you can start using it today! There is no extra charge for the compression; your CloudFront data transfer charges may actually go down (the specifics depend on the proportion of compressed to uncompressed requests, of course).

AWS Marketplace is an online store that helps you to find, buy, and immediately start using a very wide variety of applications on AWS (some of the more popular categories are Network Infrastructure, Security, and Big Data).

Up until now, running an application from AWS Marketplace was essentially equivalent to launching a single, self-contained Amazon EC2 instance. This was a good starting point, but it was not sufficient to deal with more sophisticated applications that run across a cluster of instances and/or require additional AWS resources such as Auto Scaling groups, Elastic Load Balancers, SQL database instances, an advanced network configuration, message queues, and so forth.

Support for Clusters and AWS Resources
To address this customer need, an application in the AWS Marketplace can now be represented by up to three AWS CloudFormation templates, each created by the application vendor, and each with a distinct set of deployment options. Before you actually launch a template-backed product, you will see a list of the AWS resources that will be created, along with an estimate of the monthly costs. Vendors also have the option to provide the traditional AMI-powered option alongside the new and more powerful template-powered options.

Initial Application Support
We have been working with a group of application vendors to make their applications available in this new and more flexible fashion. Here’s what you can launch today:

• Sophos –Sophos UTM 9 Autoscaling.
• Tibco – Jaspersoft for AWS with Multi-Tenancy (Hourly).
• NetApp – OnCommand Cloud Manager on Linux.
• MapR – Enterprise Database Edition Plus.

We’ll be adding more applications in the very near future.

As you can see from this screen shot, the deployment options are clearly visible, as is the estimated cost:

Template Power!
Because this new option makes use of CloudFormation, sophisticated users have access to some interesting new features. The launch process makes use of the CloudFormation console and supports the usual prompting for parameters and generation of multiple output values. Templates can be downloaded, inspected, and even edited in the CloudFormation Designer that we launched earlier this year.

Note to Application Vendors
If you are already selling your products on the Marketplace and would like to take advantage of this new option, contact your AWS BDM (Business Development Manager) or email the AWS Marketplace team at aws-marketplace-seller-ops@amazon.com. If you are new to AWS Marketplace, start by reading our Sell on Marketplace page.

You can use Amazon Virtual Private Cloud to create a logically isolated section of the AWS Cloud. Within the VPC, you can define your desired IP address range, create subnets, configure route tables, and so forth. You can also use a virtual private gateway to connect the VPC to your existing on-premises network using a hardware Virtual Private Network (VPN) connection.

An interesting network challenge arises when EC2 instances in a private VPC subnet need to connect to the Internet. Because the subnet is private, the IP addresses assigned to the instances cannot be used in public. Instead, it is necessary to use Network Address Translation (NAT) to map the private IP addresses to a public address on the way out, and then map the public IP address to the private address on the return trip.

New Managed NAT Gateway
Performing this translation at scale can be challenging. In order to simplify the task (and, as usual, to let you spend more time on your application and on your business), we are launching a new Managed NAT Gateway for AWS!

Instead of configuring, running, monitoring, and scaling a cluster of EC2 instances (you’d need at least 2 in order to ensure high availability), you can now create and configure a gateway with a couple of clicks.

The gateway has built-in redundancy for high availability. Each gateway that you create can handle up to 10 Gbps of bursty TCP, UDP, and ICMP traffic, and is managed by Amazon. You control the public IP address by assigning an Elastic IP Address when you create the gateway.

Creating a Managed NAT Gateway
Let’s create a Managed NAT Gateway! Open up the VPC Console, and take a peek at the navigation area on the left. Locate and click on NAT Gateways:

Then click on Create NAT Gateway and choose one of your subnets:

Choose one of your existing Elastic IP addresses, or create a new one:

Then click on Create a NAT Gateway, and observe the confirmation:

As you can see from the confirmation, you will need to edit your VPC’s route tables to send traffic destined for the Internet toward the gateway. The gateway’s internal (private) IP address will be chosen automatically, and will be on the subnet associated with the gateway. Here’s a sample route table:

And that’s all you need to do. You don’t need to size, scale, or manage the gateway.

You can use VPC Flow Logs to capture the traffic flowing through your gateway, and then use the information in the logs to create CloudWatch metrics based on packets, bytes, and protocols. You can use the following filter pattern as a starting point (be sure to enter actual values for ENI_ID and NGW_IP):

[version, account_id, interface_id=ENI_ID, src_addr, dst_addr=NGW_IP, src_port, dst_port, protocol, packets, bytes, start, end, action, log_status]

The resulting graph will look like this:

If you create a new VPC using the VPC Wizard, it will offer to create a NAT Gateway and the route table rules for you. This makes the setup process even easier!

To learn more, read about the VPC NAT Gateway in the VPC User Guide.

Pricing and Availability
You can start using this new feature today in the US East (Northern Virginia), US West (Oregon), US West (Northern California), Europe (Ireland), Asia Pacific (Singapore), Asia Pacific (Sydney), and Asia Pacific (Tokyo) regions.

Pricing starts at $0.045 per NAT gateway hour plus data processing and data transfer charges. Data processing costs are based on the amount of data processed by the NAT Gateway; data transfer costs are the usual costs to move data between an EC2 instance and the Internet. For more information, read about VPC Pricing.

Back when I was young, InfoWorld was a tabloid-sized journal that chronicled the growth of the PC industry. Every week I would await the newest issue and read it cover to cover, eager to learn all about the latest and greatest hardware and software. I always enjoyed and appreciated the reviews — they were unfailingly deep, objective, and helpful.

With this as background, I am really happy to be able to let you know that the team at InfoWorld recently put Amazon Aurora through its paces, wrote a detailed review, and named it an Editor’s Choice. They succinctly and accurately summarized the architecture, shared customer feedback from AWS re:Invent, and ran an intensive benchmark, concluding that:

This level of performance is far beyond any I’ve seen from other open source SQL databases, and it was achieved at far lower cost than you would pay for an Oracle database of similar power.

We’re very proud of Amazon Aurora and I think you’ll understand why after you read this review.

The AWS Trusted Advisor helps you to provision and configure your AWS resources so as to improve system performance and reliability, increase security, and optimize for cost. We have added some new checks and improved an existing one in order to make Trusted Advisor even more useful to you. Here is a summary of the changes:

The Service Limits check now reports on your usage of EC2 On-Demand instances:

This check is available to all users of Trusted Advisor. The remaining checks are available to customers who are using AWS Support API at the Business or Enterprise level.

The S3 Bucket Logging Configuration check now looks to see if server access logging has been configured for each bucket:

The new EC2 to EBS Throughput check looks for EBS volumes that might be affected by the throughput capacity of the EC2 instances:

The new CloudFront Alternate Domains check looks at the DNS settings for alternate domains on your CloudFront distributions:

The new CloudFront SSL Certificate on the Origin Server check looks for SSL certificates that are expired, about to expire, or that use outdated encryption:

The new IAM Access Key Rotation check looks for IAM keys that have not been rotated in the last 90 days:

The new checks are available now and you can benefit from them today. Visit the AWS Trusted Advisor to learn more.

When we launched EC2 Run Command seven weeks ago (see my post, New EC2 Run Command – Remote Instance Management at Scale to learn more), I promised similar functionality for instances that run Linux. I am happy to be able to report that this functionality is available now and that you can start using it today.

Run Command for Linux
Like its Windows counterpart, this feature is designed to help you to administer your EC2 instances in an easy and secure fashion, regardless of how many you are running. You can install patches, alter configuration files, and more. To recap, we built this feature to serve the following management needs:

• A need to implement configuration changes across their instances on a consistent yet ad hoc basis.
• A need for reliable and consistent results across multiple instances.
• Control over who can perform changes and what can be done.
• A clear audit path of what actions were taken.
• A desire to be able to do all of the above without the need for unfettered SSH access.

This new feature makes command execution secure, reliable, convenient, and scalable. You can create your own commands and exercise fine-grained control over execution privileges using AWS Identity and Access Management (IAM). All of the commands are centrally logged to AWS CloudTrail for easy auditing.

Run Command Benefits
The Run Command feature was designed to provide you with the following benefits (these apply to both Linux and Windows):

Control / Security – You can use IAM policies and roles to regulate access to commands and to instances. This allows you to reduce the number of users who have direct access to the instances.

Reliability – You can increase the reliability of your system by creating templates for your configuration changes. This will give you more control while also increasing predictability and reducing configuration drift over time.

Visibility – You will have more visibility into configuration changes because Run Command supports command tracking and is also integrated with CloudTrail.

Ease of Use – You can choose from a set of predefined commands, run them, and then track their progress using the Console, CLI, or API.

Customizability – You can create custom commands to tailor Run Command to the needs of your organization.

Using Run Command on Linux
Run Command makes use of an agent (amazon-ssm-agent) that runs on each instance. It is available for the following Linux distributions:

• Amazon Linux AMI (64 bit) – 2015.09, 2015.03, 2014.09, and 2014.03.
• Ubuntu Server (64 bit) – 14.04 LTS, 12.04 LTS
• Red Hat Enterprise Linux (64 bit) – 7.x

Here are some of the things that you can do with Run Command:

• Run shell commands or scripts
• Add users or groups
• Configure user or group permissions
• View all running services
• Start or stop services
• View system resources
• View log files
• Install or uninstall applications
• Update a scheduled (cron) task

You can launch new Linux instances and bootstrap the agent by including a few lines in the UserData like this (to learn more, read Configure the SSM Agent in the EC2 Documentation):

Here’s how I choose a command document (separate command documents are available for Linux and for Windows):

And here’s how I select the target instances and enter in a command or a set of commands to run:

Here’s the output from the command:

Here’s how I review the output from commands that I have already run:

Run a Command Today
This feature is available now and you can start using it today in the US East (Northern Virginia), US West (Oregon), and Europe (Ireland) regions. There’s no charge for the command, but you will be billed for other AWS resources that you consume.

To learn more, visit the Run Command page.

We announced the t2.nano instances earlier this year. Like their larger siblings (t2.micro, t2.small, t2.medium, and t2.large), these instances provide a baseline level of processing power, along with the ability to save up unused cycles and use them when the need arises.

As I noted in my earlier post (New T2.Large Instances), this model has proven to be extremely popular with our customers. In fact, we did some research and found that, over the course of a couple of days, over 96% of the T2 instances always maintained a positive CPU Credit balance. In effect, you are paying for a very modest amount of processing power, yet have access to far more when the need arises. The pricing (which I will get to in a moment) becomes even more compelling when you purchase a 1 year or 3 year Reserved Instance.

I expect to see the t2.nano used to host low-traffic websites, run microservices, support dev / test environments, and to be used as cost-effective monitoring vehicles. There are also plenty of ways to use these instances in training and educational settings.

The Specs
Each t2.nano instance has 512 MiB of memory and 1 vCPU, and can run 32 or 64 bit operating systems and applications. They support EBS encryption and up to two Elastic Network Interfaces per instance.

The t2.nano offers the full performance of a high frequency Intel CPU core if your workload utilizes less than 5% of the core on average over 24 hours. You get full access to the CPU core when needed, as long as you maintain a positive CPU credit balance. Each newly launched t2.nano starts out with a CPU credit balance of 30 credits, and earns 3 more credits per hour, up to a maximum of 72. This means that each instance can burst to full-core performance for up to 72 minutes at a stretch.

You can run Linux or Windows on these instances. However, our data shows that Windows instances consume more CPU and memory than Linux instances and you’ll want to do some testing and evaluation in order to decide which instance size will work best for your application. If you do not need the Windows GUI, you may want to take a look at the Server Core AMI.

EC2 Pricing & Sample Configurations
The t2.nano instances are priced at exactly half of the t2.micro for a given region. Here are some sample prices (see the EC2 Pricing page for more information):

Let’s take a look at the full-system cost to host and run a low-traffic website (up to 25,000 visits or so per month) on AWS using a t2.nano for one month.This is a real-world configuration that is more than adequate to handle the load.

In addition to the instance itself, the sample configuration includes an 8GB EBS SSD volume for storage and domain hosting with Amazon Route 53. The pricing includes 2 gigabytes of network-out traffic. In other words, this is the all-in cost to run the site on AWS. Here’s the monthly pricing in US West (Oregon):

Let’s say you really hit the jackpot and draw in 10 times as many visits as you planned for. You’ll pay less than $1 in additional Network Out charges, $0.81 to be precise. If you are running a small site and want to keep a watchful eye over your variable costs, don’t forget to create a billing alert.

This is a powerful starter system that can easily scale to handle more traffic or to host a more complex site or application. Over time, you can expand to make use of other AWS services such as S3, Elastic Load Balancing, Auto Scaling, Amazon Relational Database Service (RDS), and AWS CloudFormation. You also have access to T2 instances in other sizes, and to the full range of EC2 instance types.

Our friends at Bitnami provide a very wide range of packaged tools and applications that can be used on AWS with a couple of clicks. They have optimized their very popular WordPress AMI for use on the t2.nano. You can find this and many other applications in the AWS Marketplace.

Available Now
You can launch t2.nano instances today in the US East (Northern Virginia), US West (Oregon), US West (Northern California), Europe (Ireland), Asia Pacific (Singapore), Asia Pacific (Tokyo), South America (Brazil), and AWS GovCloud (US) regions. The instances will be available soon in Europe (Frankfurt) and Asia Pacific (Sydney). You can use them with AWS CloudFormation today; support for AWS Elastic Beanstalk (in the form of updated containers) is in the works.

— Jeff;

Here are the next batch of podcasts that I recorded on Tuesday, September 1 at the AWS Loft in San Francisco as part of the Intel Startup Spotlight for the AWS Podcast.

I spoke with startups Compgun, Primadesk, SkinnyPrice. I also spoke with Cameron Peron, a startup advisor. Here are the episodes and the show notes (the “Episode” links go directly to the MP3 files; you can also visit the AWS Podcast page and subscribe to the feed):

Episode 115 – Compgun
For Episode 115, I interviewed Jake Seip and Tim Sze, co-founders and CEO’s of sales commission engine Compgun. After realizing how difficult it can be to appropriately administer sales compensation and noticing that other companies share this burden, Jake and Tim used their industry insights to solve that problem. Listen to the guys chat about the strengths and weaknesses of running on AWS, where they see the company going in the future, and what they love most about working for a startup.

Episode 116 – Primadesk
For Episode 116, I spoke with Srinivasa Venkataraman, one of the founders of security startup Primadesk. Srinivasa discusses how a family photo-organization project inspired him to found Primadesk and how his tech background plus managerial experience has helped him to succeed in the startup ecosystem. Find out more about his file sharing tools, content indexing systems, and how they appeal to both consumers and IT professionals.

Episode 117 – SkinnyPrice
For Episode 117, I interviewed Andrew Jones and Daniel McGuire, the CTO and CEO/Co-Founder of SkinnyPrice. The guys talk about their experiences launching the discount pricing startup and their history of entrepreneurial adventures (including running a construction company and working with the MassChallenge Accelerator in Boston).

Episode 118 – Cameron Peron
For Episode 118, I spoke with startup advisor Cameron Peron about the challenges facing startups today and the approaches he takes to help startups succeed. We discussed his business-focused background and experience as the VP of Marketing and CMO of Redis Labs. Cameron shares his insights about working for a startup in Israel vs. in Silicon Valley, how to work with scarce resources, and his best tips for startup success.

Special Thanks
Once again, special thanks are due to my colleagues:

• Gloria Kim – Scheduling and hosting at the AWS Loft in San Francisco.
• Melissa Higa – Program Management.
• Sarah Silverstein – Editing and content management.

PS – I am planning to expand the scale of the AWS Podcast series in 2016. Stay tuned for information on how to apply to be a guest.