AWS Blog

Last year I wrote about AWS Device Farm and told you how you can use it to Test Mobile Apps on Real Devices. As I described at the time, AWS Device Farm allows you to create a project, identify an application, configure a test, and then run the test against a variety of iOS and Android devices.

Remote Access to Devices
Today we are launching a new feature that provides you with remote access to devices (phones and tablets) for interactive testing. You simply open a new session on the desired device, wait (generally a minute or two) until the device is available, and then interact with the device via the AWS Management Console.

You can gesture, swipe, and interact with devices in real time directly through your web browser as if the device was on your desk or in your hand. This includes installing and running applications!

Here’s a quick demo. I click on Start a new session to begin:

Then I search for a device of the desired type, including the desired OS version, select it, and name my session. I click on Confirm and start session to proceed:

Then I wait for the device to become available (about 30 seconds in this case):

Once the device is available I can see the screen and access it through the Console:

I can interact with the Kindle Fire using my mouse. Perhaps my app is not behaving as expected when the language is set to Latin American Spanish. I can change the Kindle Fire’s settings with a couple of clicks:

I can install my app on the Kindle Fire by clicking on Upload and choosing my APK.

My session can run for up to 60 minutes. After that time it will stop automatically.

Available Now
This new feature is available in beta form now, with a wide selection of Android phones and tablets. We will be adding iOS devices later this year, along with additional control over the device configuration and (virtual) location.

AWS Device Farm comes with a one-time free trial of 250 device minutes. After that you are charged $0.17 per device minute. Or you can pay $250 per slot per month for unmetered access (slots are the units of concurrent execution).

Amazon Cognito makes it easy for mobile and web apps to easily add authentication, user management, and data synchronization without having to write backend code or manage any infrastructure. In the last year we have added some powerful new features to Cognito including AWS CloudTrail support, the use of Twitter and Digits as login providers, the ability to run AWS Lambda functions in response to events in Cognito, and the streaming of user identity data into Amazon Kinesis.

Your User Pools
You can now use Amazon Cognito to easily add user sign-up and sign-in to your mobile and web apps. With the user pools feature, you can create your own user directory that can scale to hundreds of millions of users, and is fully managed so you don’t have to worry about the heavy lifting associated with building, securing, and scaling authentication to your apps. This feature also provides enhanced security functionality such as email verification, phone number verification, and multi-factor authentication. As an app developer, you already had the option to use an external identity provider such as Amazon, Facebook, Google, Twitter or Digits for this purpose using the Cognito feature that we now call Federated Identity Pools.

Using a user pool gives you detailed control over the sign-up and sign-in aspects of your web and mobile SaaS apps, games, and so forth. Building and running a directory service at scale (potentially tens or even hundreds of millions of users) is not easy, but is definitely undifferentiated heavy lifting, with the added security burden that comes when you are managing user names, passwords, email addresses, and other sensitive pieces of information. You don’t need to build or run your own directory service when you use Cognito Identity.

Multiple User Pools per Account
You can create multiple user pools within your AWS account (the identities in one pool are separate and distinct from those in any other pools). Each user pool is named, and you have full control over the attributes (address, email, phone number(s), locale, and so forth) that you want to store for each user. After you create a user pool, you can use the AWS Mobile SDKs (available for iOS, Android, and JavaScript) to authenticate users, get access credentials, and so forth.

Your users will benefit from a number of security features including SMS-based Multi-Factor Authentication (MFA) and account verification via phone or email. The password features use the Secure Remote Password (SRP) protocol to avoid sending cleartext passwords over the wire.

Creating a User Pool
Let’s walk through the process of creating a user pool from the AWS Management Console (API and CLI support is also available). My (hypothetical) app is called PaymentApp so I’ll create a pool named PaymentAppUsers:

I can either review and accept the defaults, or I can step through all of the settings. I’ll choose the latter course of action. Now I choose the set of attributes that must be collected when a new user signs up for my service:

I can also set up custom attributes. For my app, I would like to track the user’s preferred payment currency:

Then I set up the desired password strength. Since this is payment app, I’ll check all of the options:

Next, I enable Multi-Factor Authentication, and indicate that email addresses and phone numbers must be verified. I also customize the messages that are associated with each method of verification:

My app will have a mobile client so I’ll arrange to create a unique ID and a secret key for it:

Now I can arrange to have Cognito invoke some Lambda functions during the sign-up, verification, authentication, and confirmation steps (this is optional, but very useful if you want to customize the sign-up workflow by validating custom attributes):

Finally, I review my choices and create my pool:

At this point I am ready to build my web or mobile app.

Cognito at Asurion
Asurion provides over 200 million customers with insurance policies for high-value devices such as smartphones. Asurion plans to use Cognito to manage the user directory for their new device protection app. This app will collect device-related data and make recommendations that are aimed at optimizing usage.

Asurion chose Cognito due to its support for a wide variety of identity models. They can have their own fully managed directory or they can have users authenticate through third-party social identity providers without having to deal with the heavy lifting involved in scaling and security an identity system.

Ravi Tiyyagura (Asurion’s Director of Enterprise Architecture) told us:

It is critical for us to provide a secure and simple sign-up and sign-in experience for our tens of millions of end users. With Amazon Cognito, we can enable that without having to worry about building and managing any backend infrastructure.

Public Beta
We are launching user pools today as a public beta. All of the primary functionality is in place and you can use it to start building and testing your app. We expect to make it available for production use within a couple of months.

Several AWS teams are focused on simplifying and accelerating the process of moving on-premises data to the cloud. We started out with the very basic PUT operation and multipart upload in the early days. Along the way we gave you the ability to send us a disk, and made that process even easier by launching the AWS Import/Export Snowball at last year’s AWS re:Invent (read AWS Import/Export Snowball – Transfer 1 Petabyte Per Week Using Amazon-Owned Storage Appliances to learn more).

Today we are launching some important improvements to Amazon S3 and to Snowball, both designed to further simplify and accelerate your data migration process. Here’s what’s new:

Amazon S3 Transfer Acceleration – This new feature accelerates Amazon S3 data transfers by making use of optimized network protocols and the AWS edge infrastructure. Improvements are typically in the range of 50% to 500% for cross-country transfer of larger objects, but can go ever higher under certain conditions.

Larger Snowballs in More Regions – A new, larger-capacity (80 terabyte) Snowball appliance is now available. In addition, the appliances can now be used with two additional US Regions and two additional international Regions.

Amazon S3 Transfer Acceleration
The AWS edge network has points of presence in more than 50 locations. Today, it is used to distribute content via Amazon CloudFront and to provide rapid responses to DNS queries made to Amazon Route 53. With today’s announcement, the edge network also helps to accelerate data transfers in to and out of Amazon S3. It will be of particular benefit to you if you are transferring data across or between continents, have a fast Internet connection, use large objects, or have a lot of content to upload.

You can think of the edge network as a bridge between your upload point (your desktop or your on-premises data center) and the target bucket. After you enable this feature for a bucket (by checking a checkbox in the AWS Management Console), you simply change the bucket’s endpoint to the form BUCKET_NAME.s3-accelerate.amazonaws.com. No other configuration changes are necessary! After you do this, your TCP connections will be routed to the best AWS edge location based on latency.  Transfer Acceleration will then send your uploads back to S3 over the AWS-managed backbone network using optimized network protocols, persistent connections from edge to origin, fully-open send and receive windows, and so forth.

Here’s how I enable Transfer Acceleration for my bucket (jbarr-public):

My bucket endpoint is https://jbarr-public.s3.amazonaws.com/. I simply change it to https://jbarr-public.s3-accelerate.amazonaws.com/ in my upload tool and/or code. Then I initiate the upload and take advantage of S3 Transfer Acceleration with no further effort. I can use the same rule to construct a URL that can be used for accelerated downloads.

Because network configurations and conditions vary from time to time and from location to location, you pay only for transfers where Transfer Acceleration can potentially improve upload performance. Pricing for accelerated transfers begins at $0.04 per gigabyte uploaded. As is always the case with S3, there’s no up-front fee or long-term commitment.

You can use our new Amazon S3 Transfer Acceleration Speed Comparison to get a better understanding of the value of Transfer Acceleration in your environment. I ran it from my Amazon WorkSpace in the US West (Oregon) Region and this is what I saw:

As you can see, the opportunity for improvement grew in rough proportion to the distance between my home Region and the target.

To learn more about this feature, read Getting Started with Amazon S3 Transfer Acceleration in the Amazon S3 Developer Guide. It is available today in all Regions except Beijing (China) and AWS GovCloud (US).

Larger Snowballs in More Regions
Many AWS customers are now using AWS Import/Export Snowball to move large amounts of data in and out of the AWS Cloud. For example, here’s a Snowball on-site at GE Oil & Gas:

Ben Wilson (CTO of GE Oil & Gas) posted the picture to LinkedIn with the following caption:

“PIGs and Snowballs – a match made in heaven!! AWS Snowball 25 TB of Pipeline PIG Data to be managed at AWS. That is our GE PIG we pulled some of the data from. It’s always fun to try new AWS features and try to break them!!”

Today we are making Snowball available in four new Regions: AWS GovCloud (US), US West (Northern California), Europe (Ireland), and Asia Pacific (Sydney). We expect to make Snowball available in the remaining AWS Regions in the coming year.

The original Snowball appliances had a capacity of 50 terabytes. Today we are launching a newer appliance with 80 terabytes of capacity. If you are transferring data in or out of the US East (Northern Virginia), US West (Oregon), US West (Northern California), or AWS GovCloud (US) Regions using Snowball you can choose the desired capacity. If you are transferring data in or out of the Europe (Ireland) or Asia Pacific (Sydney) Regions, you will use the 80 terabyte appliance. Here’s how you choose the desired capacity when you are creating your import job:

To learn more about Snowball, read the Snowball FAQ and the Snowball Developer Guide.

Amazon Kinesis makes streaming data easy in the cloud.The Amazon Kinesis platform is comprised of three distinct services: Kinesis Streams allows developers to build their own stream processing applications; Kinesis Firehose simplifies the process of loading streaming data into AWS for storage and analytics; Kinesis Analytics supports the analysis of streaming data using standard SQL queries.

Many AWS customers use Kinesis Streams and Kinesis Firehose as a component of their real-time streaming data ingestion and processing systems. They appreciate the ease of use that comes with a fully managed service, and invest their development time in their application instead of spending time managing their own streaming data infrastructure.

Today we are announcing three new features for Amazon Kinesis Streams and Amazon Kinesis Firehose:

• Elasticsearch Integration – Amazon Kinesis Firehose can now stream data to an Amazon Elasticsearch Service cluster.
• Enhanced Metrics – Amazon Kinesis now sends shard-level metrics to CloudWatch each minute.
• Flexibility – Amazon Kinesis now allows you to retrieve records using time-based shard iterators.

Amazon Elasticsearch Service Integration
Elasticsearch is a popular open-source search and analytics engine. Amazon Elasticsearch Service is a managed service that makes it easy for you to deploy, run, and scale Elasticsearch in the AWS Cloud. You can now arrange to deliver your Kinesis Firehose data stream to an Amazon Elasticsearch Cluster. This will allow you to index and analyze server logs, clickstreams, and social media traffic.

The incoming records (Elasticsearch documents) are buffered in Kinesis Firehose according to a configuration that you specify, and then automatically added to the cluster using a bulk request that indexes multiple documents simultaneously. The data must be UTF-8 encoded and flattened into single JSON object before it is sent to Firehose (see my recent blog post, Amazon Kinesis Agent Update – New Data Preprocessing Feature, to learn more about how to do this).

Here’s how to set this up using the AWS Management Console. I choose the destination (Amazon Elasticsearch Service) and set the delivery stream name, then I choose one of my Elasticsearch domains (livedata in this example), set up the index, and choose the index rotation (none, hourly, daily, weekly, or monthly). I also designate an S3 bucket that will receive a backup of either all documents or failed documents (my choice):

Then I set the buffer size, choose some compression and encryption options for the data that will be sent to my S3 bucket, set up logging (if desired), and pick an appropriate IAM role:

The stream will be ready for use in a minute or so:

I can view the delivery metrics in the Console:

Once the data starts to arrive in Elasticsearch I can explore it visually using Kibana or by writing queries in the Elasticsearch query language.

Putting this all together, this integration greatly simplifies the process of capturing and delivering your streaming data to your Elasticsearch cluster. There’s no need to write any code or to build your own data ingestion tools.

Shard-Level Metrics
Each Kinesis stream is composed of one or more shards, each of which provides a fixed amount of read and write capacity. Each time you add a shard to a stream, you increase the capacity of the stream.

In order to provide you with increased visibility into the performance of each shard, you can now enable a set of shard-level metrics. There are 6 metrics per shard, each reported once per minute and charged at the usual per-metric CloudWatch pricing. These metrics will allow you to see if a particular shard is running hotter than the others and to locate and root out any inefficiencies in your end-to-end streaming data delivery pipeline. For example, you can identify the shard(s) that are receiving records at a rate too high too handle and the shard(s) that are being read by applications at lower throughput than expected.

Here are the new metrics:

IncomingBytes – The number of bytes that have been successfully PUT to the shard.

IncomingRecords – The number of records that have been successfully PUT to the shard.

IteratorAgeMilliseconds – The age (in milliseconds) of the last record returned by a GetRecords call against a shard. A value of 0 means that the records being read are completely caught up with the stream.

OutgoingBytes – The number of bytes that have been retrieved from the shard.

OutgoingRecords – The number of records that have been retrieved from the shard.

ReadProvisionedThroughputExceeded -The number of GetRecords calls that have been throttled for exceeding the 5 reads per second or 2 MB per second shard limits.

WriteProvisionedThroughputExceeded – The number of records that have been rejected due to throttling for exceeding the 1000 records per second or 1 MB per second shard limits.

You can enable these metrics by calling the EnableEnhancedMonitoring function. As always, you can use the CloudWatch APIs to aggregate them across any desired time period.

Time-Based Iterators
Your application reads data from a Kinesis stream by creating an iterator on the desired shard using the GetShardIterator function and specifying the desired starting point. In addition to the existing starting point options (at or after a sequence number, oldest record, or newest record) you can now specify a timestamp. The value (specified in Unix epoch format) indicates the timestamp of the oldest record that you would like to read and process.

AWS Elastic Beanstalk simplifies the process of deploying and running web applications and web services. You simply upload your code and Elastic Beanstalk will take care of the details. This includes provisioning capacity, setting up load balancing and auto scaling, and arranging for application health monitoring. You can build Elastic Beanstalk applications using a variety of platforms and languages including Java, PHP, Ruby, Node.ps, Python, .NET, Go, and Docker.

Elastic Beanstalk regularly releases new versions of supported platforms with operating system, web & app server, and language & framework updates. Until now, you needed to initiate a manual update (via the Elastic Beanstalk Console, command line interface, or API) to update your Elastic Beanstalk environments to the new version of the platform or language.  This gave you full control over the timing of updates, but left you with one more thing to remember and to manage.

Managed Platform Updates
Today we are making Elastic Beanstalk even more powerful by adding support for managed platform updates. You simply select a weekly maintenance window and Elastic Beanstalk will update your environment to the latest platform version automatically.

The updates are installed using an immutable  deployment model to ensure that no changes are made to the existing environment until the updated replacement instances are available and deemed healthy (according to the health check that you have configured for the application). If issues are detected during the update, traffic will continue to be routed to the existing instances. The immutable deployment model also ensures that your application will remain available during updates in order to minimize disruption to your users.

You can choose to install minor updates and patches automatically, and you can also trigger updates outside of the maintenance window. Because major updates typically require careful testing before being deployed, they will not take place automatically and must be triggered manually.

You can configure managed updates from the Elastic Beanstalk Console. First, enable them in the Configuration tab:

And then manage them in the Managed Updates tab:

Available Now
This new feature is available now and you can start using it today. There’s no charge for the feature, but you will pay for any additional EC2 instances that are used to ensure a seamless update.

The AWS team spends a lot of time looking in to ways to deliver innovation based around improvements in price/performance. Quite often, this means wrestling with interesting economic and technical dilemmas.

For example, it turns out that there are some really interesting trade-offs between HDD and SSD storage. On the one hand, today’s SSD devices provide more IOPS per dollar, more throughput per gigabyte, and lower latency than today’s HDD devices. On the other hand, continued density improvements in HDD technology drive the cost per gigabyte down, but also reduce the effective throughput per gigabyte. We took this as a challenge and asked ourselves—could we use cost-effective HDD devices to build a high-throughput storage option for EBS that would deliver consistent performance for common workloads like big data and log processing?

Of course we could!

Today we are launching a new pair of low-cost EBS volume types that take advantage of the scale of the cloud to deliver high throughput on a consistent basis, for use with EC2 instances and Amazon EMR clusters (prices are for the US East (Northern Virginia) Region; please see the EBS Pricing page for other regions):

• Throughput Optimized HDD (st1) – Designed for high-throughput MapReduce, Kafka, ETL, log processing, and data warehouse workloads; $0.045 / gigabyte / month.
• Cold HDD (sc1) – Designed for workloads similar to those for Throughput Optimized HDD that are accessed less frequently; $0.025 / gigabyte / month.

Like the existing General Purpose SSD (gp2) volume type, the new magnetic volumes give you baseline performance, burst performance, and a burst credit bucket. While the SSD volumes define performance in terms of IOPS (Input/Output Operations Per Second), the new volumes define it in terms of throughput. The burst values are based on the amount of storage provisioned for the volume:

• Throughput Optimized HDD (st1) – Starts at 250 MB/s for a 1 terabyte volume, and grows by 250 MB/s for every additional provisioned terabyte until reaching a maximum burst throughput of 500 MB/s.
• Cold HDD (sc1) – Starts at 80 MB/s for a 1 terabyte volume, and grows by 80 MB/s for every additional provisioned terabyte until reaching a maximum burst throughput of 250 MB/s.

Evolution of EBS
I like to think of customer-driven product and feature development in evolutionary terms. New offerings within a category often provide broad solutions that are a good fit for a wide variety of use cases. Over time, as we see how customers put the new offering to use and provide us with feedback on how we can do even better, a single initial offering will often speciate into several new offerings, each one tuned to the needs of a particular customer type and/or use case.

The various storage options for EC2 instances are a great example of this. Here’s a brief timeline of some of the most significant developments:

• 2006 – EC2 launched with instance storage.
• 2008 – EBS (Elastic Block Storage) launched on magnetic storage.
• 2012 – EBS Provisioned IOPS and EBS-Optimized instances.
• 2014 – SSD-Backed general purpose storage.
• 2014 – EBS data volume encryption.
• 2015 – Larger and faster EBS volumes.
• 2015 – EBS boot volume encryption.
• 2016 – EBS Throughput Optimized HDD (st1) and Cold HDD (sc1) volume types.

Workload Characteristics
We tuned these volumes to deliver great price/performance when used for big data workloads. In order to achieve the levels of performance that are possible with the volumes, your application must perform large and sequential I/O operations, which is typical of big data workloads. This is due to the nature of the underlying magnetic storage, which can transfer contiguous data with great rapidity. Small random access I/O operations (often generated by database engines) are less efficient and will result in lower throughput. The General Purpose SSD volumes are a much better fit for this access pattern.

For both of the new magnetic volume types, the burst credit bucket can grow until it reaches the size of the volume. In other words, when a volume’s bucket is full, you can scan the entire volume at the burst rate. Each I/O request of 1 megabyte or less counts as 1 megabyte’s worth of credit. Sequential I/O operations are merged into larger ones where possible; this can increase throughput and maximizes the value of the burst credit bucket (to learn more about how the bucket operates, visit the Performance Burst Details section of my New SSD-Backed Elastic Block Storage post).

If your application makes use of the file system and the operating system’s page cache (as just about all applications do), we recommend that you set the volume’s read-ahead buffer to 1 MiB on the EC2 instance that the volume is attached to. Here’s how you do that using an instance that is running Ubuntu or that was booted from the Amazon Linux AMI (adjust the device name as needed):

$ sudo blockdev --setra 2048 /dev/xvdf

The value is expressed as the number of 512-byte sectors to be used for buffering.

This value will improve read performance for workloads that consist of large, sequential reads. However, it may increase latency for workloads that consist of small, random read operations.

Most customers are using Linux kernel versions before 4.2 and the read ahead setting is all they need to tune. For customers using newer kernels, we also recommend setting xen_blkfront.max to 256 for the best performance. To set this parameter on an instance that runs the Amazon Linux AMI, edit /boot/grub/menu.list so that it invokes the kernel as follows:

kernel /boot/vmlinuz-4.4.5-15.26.amzn1.x86_64 root=LABEL=/ console=ttyS0 xen_blkfront.max=256

If your file contains multiple entries, edit the one that corresponds to the active kernel.  This is a boot-time setting so you’ll need to reboot the instance in order for the setting to take effect. If you are using a Linux distribution that does not use the Grub bootloader, you will need to figure out how to make the equivalent change to your configuration.

For more performance tuning tips, please read Amazon EBS Volume Performance on Linux Instances and Amazon EBS Volume Performance on Windows Instances.

Comparing EBS Volume Types
Here’s a table that summarizes the specifications and use cases of each EBS volume type (Although not shown in the table, the original EBS Magnetic offering is still available if needed for your application):

You also have the option to further boost performance by using EBS-Optimized instances and RAID to create file systems that are larger and/or support more IOPS. Read about RAID Configuration on Linux and RAID Configuration on Windows to learn more.

CloudFormation Template for Testing
In order to make it as easy as possible for you to set up a test environment on a reproducible basis, we have created a simple CloudFormation template. You can launch the st1 template to create an EC2 instance with a 2 terabyte st1 volume attached. The st1 template instructions contain some additional information.

From our Partners
Several AWS partners have been working with the new volume types ahead of today’s launch. Here are their thoughts and observations:

• Localytics shared their Amazon EBS SC1 Initial Impressions. They created a petabyte scale data warehouse backed by a RAIDed set of sc1 volumes. After running for a day, they saw that performance was indistinguishable from that provided by their earlier configuration. Based on their calculations the new configuration will allow them to save 10% on their platform operating costs.
• Confluent wrote about Deploying Apache Kafka on AWS Elastic Block Store (EBS). After taking a look at the new offerings, they are looking forward to creating Kafka clusters that are more cost-effective than ever before.
• Splunk explained how their customers can Retain More Data at Lower Cost with new AWS Storage Volume Types. According to the cost model described in the blog post, Splunk users can achieve a 71% savings by moving some of their colder data to sc1 storage.

Available Now
The new volume types are available now and you can start using them today with EC2 and EMR. You can create them from the AWS Management Console, AWS Command Line Interface (CLI), AWS Tools for Windows PowerShell, AWS CloudFormation templates, the AWS SDKs, and so forth.

As you can see from the table above, this new offering gives you a unique combination of high throughput and a very low cost per gigabyte.

I am looking forward to your feedback so that we can continue to evolve EBS to meet your ever-growing (and continually diversifying) needs. Leave me a comment and I’ll make sure that the team sees it.

PS – If you are a developer, development manager, or product manager and would like to build systems like this, please visit the EBS Jobs page.

Our friends over at Box provide secure content management, collaboration, and file sharing for over half of the companies on the Fortune 500 list.

Box has succeeded by paying attention to the needs of enterprise customers. For example, last year I wrote about Box Enterprise Key Management (EKM), a flexible, no-compromises encryption system that gives Box customers control over their encryption keys. This feature has evolved into Box KeySafe, which allows even the smallest IT shops use encryption to protect their proprietary documents. Other Box features such as Box Capture (mobile phone integration with business processing) and Box Governance (control and compliance) are also purpose-built to meet the business needs of enterprises.

We are happy to play a strong supporting role in today’s launch of Box Zones. This new feature uses Amazon S3 to provide Box customers with a choice of four storage locations (Germany, Ireland, Singapore, and Tokyo). Box customers can decide where to store their data while still taking advantage of other Box features such as watermarking, fine-grained control over permissions, commenting, and file preview.

To learn more about this new feature, sign up for the Box Zones webinar.

Congratulations to Box, and thank you for using AWS!

Let’s take a quick look at what happened in AWS-land last week:

Upcoming Events

• AWS Partner Webinars – April.
• April 29 – Live Event (Singapore) – AWS Partner Summit.
• AWS Zombie Microservices Roadshow.

Help Wanted

• AWS Careers.

Stay tuned for next week! In the meantime, follow me on Twitter and subscribe to the RSS feed.

— Jeff;

Let’s take a quick look at what happened in AWS-land last week:

Stay tuned for next week! In the meantime, follow me on Twitter and subscribe to the RSS feed.

— Jeff;

Let’s take a quick look at what happened in AWS-land last week:

Stay tuned for next week! In the meantime, follow me on Twitter and subscribe to the RSS feed.

— Jeff;