AWS Official Blog

AWS Week in Review – February 15, 2016

by Jeff Barr | on | in Week in Review | | Comments

Let’s take a quick look at what happened in AWS-land last week:

Monday

February 15

Tuesday

February 16

Wednesday

February 17

Thursday

February 18

Friday

February 19

Saturday

February 20

Sunday

February 21

New & Notable Open Source

  • aws-api-gateway-for-cloudformation is a set of Custom Resources that enables API Gateway support for CloudFormation.
  • WaterFlow is a is a non-magical / easy to understand / JDK8 framework for use with Amazon Simple Workflow.
  • gnu-mailman-aws documents the process of installing and running GNU Mailman on an EC2 instance.
  • spa-aws is a single page application using AWS Lambda.
  • petit is a URL shortener written in Ruby for use on AWS.
  • state-of-cloud is a tool to inventory cloud and report of the use of AWS and other services.
  • aws-helpers is a set of Node.JS helpers for AWS.
  • aws-lambda-canary is a canary project for Lambda services on AWS.
  • cookiecutter-lambder is a cookiecutter template for creating Lambder projects.
  • aws provides Racket support for AWS projects.

New SlideShare Presentations

New Customer Success Stories

New YouTube Videos

Upcoming Events

Help Wanted

Stay tuned for next week! In the meantime, follow me on Twitter and subscribe to the RSS feed.

Jeff;

Amazon RDS Update – Support for MySQL 5.7

by Jeff Barr | on | in Amazon RDS | | Comments

You can now launch Amazon RDS database instances that run MySQL 5.7 .

This release of MySQL offers a number of performance, scalability, and security enhancements. Here are some of the most important and relevant ones:

  • Native support for JSON data and a set of built-in JSON functions (JSON_ARRAY, JSON_OBJECT, JSON_QUOTE, JSON_CONTAINS, JSON_CONTAINS_PATH, JSON_EXTRACT, JSON_KEYS, JSON_SEARCH, JSON_APPEND, JSON_ARRAY_APPEND, JSON_ARRAY_INSERT, JSON_INSERT, JSON_MERGE, JSON_REMOVE, JSON_REPLACE, JSON_SET, JSON_UNQUOTE, JSON_DEPTH, JSON_LENGTH, JSON_TYPE, and JSON_VALID).
  • A Performance Schema that provides access to new and improved performance metrics.
  • Optimizer improvements for better parsing, EXPLAINing, and querying.
  • GIS with native InnoDB spatial indexes and integration with Boost.Geometry (read MySQL 5.7 and GIS, an Example and Making Use of Boost Geometry in MySQL GIS to learn more).
  • Improved parallel replication using a new logical clock mode (read Multi-threaded Replication Performance in MySQL 5.7 to learn more).
  • Improved InnoDB scalability and temporary table performance. Improved tablespace discovery during crash recovery, and dynamic buffer pool resizing.

Read the MySQL 5.7 Release Notes for more information!

Launching a Database Instance
As always, you can launch these instances from the AWS Management Console, AWS Command Line Interface (CLI), AWS Tools for Windows PowerShell, the RDS API (CreateDBInstance), or via a CloudFormation template. Here’s how you launch a database instance from the Console:

After I launched my instance, I edited its security group to include the public IP address of one of my EC2 instances. Then I connected with it in the usual way:

Then I took a quick look at the new Performance Schema:

Time was kind of tight and my MySQL is kind of rusty so I didn’t have a chance to exercise any of the new features. I’ll leave that up to you!

Available Now
Amazon RDS for MySQL is available in the US East (Northern Virginia), US West (Northern California), US West (Oregon), Europe (Ireland), Europe (Frankfurt), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Tokyo), Asia Pacific (Seoul), China (Beijing), South America (Brazil), and AWS GovCloud (US) Regions.

Jeff;

 

 

Announcing the AWS Pop-Up Loft in Tel Aviv

by Jeff Barr | on | in AWS Loft | | Comments

The AWS Pop-up Lofts in San Francisco and New York have become hubs and working spaces for developers, entrepreneurs, students, and others who are interested in working with and learning more about AWS. They come to learn, code, meet, collaborate, ask questions, and to hang out with other like-minded folks.

I am happy to announce that we will be popping up a loft in Tel Aviv. As always, we have created a unique space and assembled a full calendar of events. Like the other lofts, this one will be a great place to meet and to connect with our customers.

Near the Startup Community
This loft is located in the heart of Tel Aviv’s startup community: Lilienblum 23. It will open for business at 10:00 on Tuesday, March 1st. After that it will be open from 10:00 to 18:00 Sunday through Thursday until March 21st.

Loft Activities & Resources
The loft will host over 100 technical and business sessions, brought to you by 50 AWS professionals. Here’s what we have in store for the first week:

  • Tuesday, March 1 – 15:30-16:30 – Introduction to AWS IoT.
  • Tuesday, March 1 – 16:00 – 17:00 – Deploy Your Apps with AWS Elastic Beanstalk.
  • Wednesday, March 2 – 16:00-17:00 – Introduction to Elastic MapReduce.
  • Sunday, March 6 – 14:00-15:00 – Deep Dive: AWS Direct Connect and VPNs.

We’ll be adding even more sessions and activities in the weeks to come.

You can also attend some of our hands-on labs and set up 1:1 meetings with members of our Customer Support, Solution Architecture, and Activate teams. To learn more, visit the Tel Aviv Loft page.

Jeff;

 

New – Notifications for AWS CodeDeploy Events

by Jeff Barr | on | in AWS CodeDeploy | | Comments

AWS CodeDeploy is a service that helps you to deploy your code to a fleet of EC2 or on-premises instances while taking care to leave as much of the fleet online as possible. CodeDeploy was designed to work with fleets that range in size from a single instance all the way up to thousands of instances (see my post, New AWS Tools for Code Management and Deployment for more information).

Notifications for CodeDeploy
In order to make it easier for you to use CodeDeploy as a part of your overall build, test, and deployment pipeline, we are introducing a new notification system today. You can now create triggers that send Amazon SNS notifications before, during, and after the deployment process for your applications. Triggers can be set for the deployment as a whole or for the individual instances targeted by the deployment, and are sent on both successes and failures. Here is the full list of triggers:

  • DEPLOYMENT_START
  • DEPLOYMENT_SUCCESS
  • DEPLOYMENT_FAILURE
  • DEPLOYMENT_STOP
  • INSTANCE_START
  • INSTANCE_SUCCESS
  • INSTANCE_FAILURE

You can create up to 10 triggers per application. You can connect several triggers to a single topic, or you can send each trigger to a distinct topic. You can use any of the delivery protocols supported by SNS (http, https, email, SMS, and mobile push). You can also invoke a Lambda function.

Creating a Trigger
Triggers can be created using the AWS Management Console or the AWS Command Line Interface (CLI). I’ll use the Console in the post. I set up the sample CodeDeploy application (three t2.micro instances):

And then I did an initial deployment:

Then I created an SNS topic, subscribed to it via email, and confirmed the subscription:

I returned to the Console and opened up my application within CodeDeploy:

I opened up IAM in another tab and updated the policy associated with the Service Role, giving it permission to write to SNS (you won’t need to do this if you select the managed AWSCodeDeployRole for your application):

Back in CodeDeploy, I clicked on Create trigger, entered a name, chose my events (all of them for this example), and selected my SNS topic from the dropdown:

I could also have chosen individual events:

My trigger was created and displayed in the Console:

CodeDeploy sent a confirming message to the topic and it was at the top of my Inbox:

Then I initiated a deployment and waited for the emails to arrive! Here’s what they looked like:

I used email for illustrative purposes; in a real-world application you would probably want to write some code to catch and handle the messages.

Triggers are available now and you can start using them today!

Jeff;

 

 

US Veterans Now Eligible For AWS Certification Exam Reimbursement

by Jeff Barr | on | in Training and Certification | | Comments

Service members returning from active duty can face many challenges—including finding a new job or even a new career. To help veterans succeed, we’ve worked with the Department of Veterans Affairs to make AWS Certification exams eligible for VA reimbursement under the GI Bill’s education provision.

About AWS Certifications
AWS Certifications recognize IT professionals with the technical expertise to design, deploy, and operate applications and infrastructure on AWS. Career transitions are never easy, but cloud IT presents one promising path forward—especially when industry surveys show that these skills remain in short supply.

Our hope is that easier access to our certification exam, combined with the unique talents veterans already possess, will open up more career possibilities for retired servicemen and servicewomen and help them achieve success in their post-military careers.

How it Works
Qualifying US veterans covered under a GI Bill with an education provision can now submit a reimbursement request to the Department of Veterans Affairs for exams taken after December 10, 2015 and purchased from Webassessor. The VA will cover exam fees up to $2000 (costs connected with preparing for a certification such as training courses or practice exams are not reimbursable).

Learn more about the benefit—or explore all AWS Training courses, videos, labs, and certifications.

Jeff;

 

Migrating to Amazon Aurora: The View from the Other Side

by Jeff Barr | on | in Amazon Aurora, Guest Post | | Comments

Today I have a new guest post, this one written by AWS customer Safe Software. They were an early adopter of Amazon Aurora and have a great story to tell.

Jeff;

Being in the business of moving data, Safe Software is always working to support leading edge database technology. This focus naturally led us to us being aware of Amazon Aurora.

In addition to leveraging the power of AWS and Aurora for our clients, we also evaluate new technologies from the perspective of improving our internal processes. When Aurora was released in beta, we immediately thought of migrating to it for our own systems. That decision has proven to be worthwhile. The move to Aurora has increased our productivity while providing an annual 40% cost reduction in systems costs.

Now that the migration is behind us, I’d like to share some tips and insights to those considering taking the leap. While I’d like to include the migration details, there is not much to say as it only took the click of a button. Instead I will share what we tried first, how we prepared, and how we optimized our systems further once they were operating in Aurora.

Why We Needed The Cloud
To ensure high quality of our spatial data transformation technology, FME, we run a grueling automated test suite. The platform supports 365+ data formats and limitless transformations, making the automated daily testing demanding: 15,000 tests x 4 operating systems x 3 products, running 24/7.

The problem: We couldn’t keep provisioning hardware to keep the system happy. Never mind keeping up with our expectation of a 1-second response time.

Our internal production database runs on a high traffic system with 140+ tables containing ~100 million rows of data. It is the primary operational repository for our build and test systems, as well as our internal intranet and reporting framework, supporting our server farms upwards of 150 machines. Over 100 users rely on this system.

What We Tried Before Aurora
We initially tried moving everything to MySQL on RDS, but found that we needed to run a read replica on a sizable instance to manage load. Even still, we were crowding ourselves against the productivity ceiling for the number of connections we could handle for most queries. We struggled to meet the needed response times. This solution had also immediately doubled our costs.

We’d spent so much time getting good at MySQL that the idea of having to relearn all of that in a new system was painful. Having something you treat like an appliance is much better.

Fail-Safe Preparations and Migration
We heard Aurora mirrors the MySQL experience, so we figured it was worth trying. To ensure we had nothing to lose we decided to keep the production system running in its existing form, while we tested Aurora.

One of the benefits of moving to a higher performance system is you have a good opportunity to re-assess a system that dates back years. During this migration we did some housekeeping. We looked at indexes, table structures, and many other relational aspects of the database. We were able to combining multiple schemas into just two for simpler logic.

The actual move into Aurora was trivial. Within the Amazon control panel, we chose to migrate – clicking a button. It happened in the background, and we ended up with a copy in Aurora of what we had running in MySQL! It was that simple!

Managing the cutover is a fairly big thing in terms of scheduling, to make sure we’re not impacting operations and meanwhile capturing a current snapshot of the data. We were wary that things could get out of sync; that by the time the migration was done the read dates may be out of date. We knew it would take a few hours to migrate the production system that was still operating, and during that time, data could change.

We chose to do the migration overnight on the live system while it was still running. We followed up with our own FME product to capture changes that had taken place in volatile tables during the migration (about 2-3% of our data), and port them over.

Our build and release team was able to manage the migration ourselves, and only involved the IT department to configure identity and access management and then change the DNS on our network once we’d verified that everything was a go.

We had checked a few examples for sanity first, but it was kind of a leap in the dark because we were early adopters. But we knew that we could just roll back to the old system if needed.

Optimizing The Experience Post-Migration
We thoroughly tested all of our processes afterward. There was some head-scratching after the first couple of days of monitoring; we experienced patches of heavy CPU load and slow-downs in Aurora during operations that had previously been unremarkable in MySQL.

We tracked these down to a set of inefficient queries using deeply nested SELECTs which were not readily modifiable. We resolved these issues with some simple schema changes, and pre-canning some of the more complex relationships using our own product, FME. Bottom Line: Schema design is still important.

No other issues were experienced during or since, and tuning these queries and indexes was ultimately beneficial. In operation we now have enterprise scale with the familiar interfaces we are used to.

For almost all operations, Aurora has proven faster, and it gives us more scalability. We’re running a relatively modest setup now, knowing that we can expand, and it’s saving us about $8,000 per year (60% cheaper). In fact, we could double our performance using Aurora and it would still be less than we paid last year. We save another $2,000 by reserving the instance for annual operations.

Operation management is pretty critical stuff, so it’s a relief not to worry about backups or database failures. The managed database saves us a lot of headaches. To achieve this performance ourselves would have a required huge investment in both hardware and personnel.

With Aurora, we can create our FME product builds better, faster, and the test results come through quickly, which ultimately means we can provide a higher quality product.

Iain McCarthy, Product Release Manager at Safe Software Inc.

Amazon EMR Update – Support for EBS Volumes, and M4 & C4 Instance Types

by Jeff Barr | on | in Amazon Elastic Block Store, Amazon EMR | | Comments

My colleague Abhishek Sinha wrote the guest post below to tell you about the latest additions to Amazon EMR.

Jeff;

Amazon EMR is a service that allows you to use distributed data processing frameworks such as Apache Hadoop, Apache Spark and Presto to process data on a managed cluster of EC2 instances.

Newer versions of EMR (3.10 and 4.x), allow you to use Amazon EBS volumes to increase the local storage of each instance. This works well with the existing set of supported instance types, and also gives you the ability to use the M4 and C4 instance types with EMR. Today I would like to tell you more about both of these features.

Increasing Instance Storage Using Amazon EBS
EMR uses the local storage of each instance for HDFS (Hadoop Distributed File System) and to store intermediate files when processing data from S3 using EMRFS. You can now use EBS volumes to extend this storage. The EBS volumes are tied to the lifecycle of the associated instances and augment any existing storage on the instance. If you terminate a cluster, any associated EBS volumes are also deleted along with it.

You will benefit from the ability to customize the storage of your EMR instances if…

  1. Your processing requirements demand a larger amount of HDFS (or local) storage than what is available by default on an instance. With support for EBS volumes, you will be able to customize the storage capacity on an instance relative to the compute capacity that the instance provides. Optimizing the storage on an instance will allow you to save costs.
  2. You want to take advantage of the latest generation EC2 instance types such as the M4, C4, and R3 and need more storage than is available on these instance types. You can now add EBS volumes to customize the storage in order to better meet your needs. If you’re using the older M1 and M2 instances, you should be able to reduce costs and improve performance by moving to  newer M4, C4 and R3 instances. We recommend that you benchmark your application to measure the impact on your specific workloads.

It’s important to note that the EBS volumes added to an Amazon EMR cluster do not persist data after the cluster is shutdown. EMR will automatically clean up the volumes when you terminate your cluster.

Adding EBS Volumes to a Cluster
EMR currently groups the nodes in your cluster into 3 logical instance groups: a Master Group, which runs the YARN Resource Manager and the HDFS Name Node Service; a Core Group, which runs the HDFS DataNode Daemon and the YARN Node Manager Service; and Task Groups, which run the YARN Node Manager Service. EMR supports up to 50 instance groups per cluster and allows you to select an instance type for each group. You can now specify the amount of EBS storage you want to add to each instance in a given instance group. You can specify multiple EBS volumes, add EBS volumes to instances with instance storage, or even combine different volumes of different types. Here is how you specify your storage configuration in the EMR Console:

For example, if you configured a Core Group to use the m4.2xlarge instance, attached a pair of 1 TB gp2 (General Purpose SSD) volumes and want 10 instances in the group, the Core group would have 10 instances with a total of 20 volumes. Here’s how you would set that up:

To learn more, read the EBS FAQ. Support for EBS is available starting with AMI 3.10 and EMR Release 4.0.

EBS Volume Performance Characteristics
Amazon EMR allows you to use several different EBS volume types: General Purpose SSD (GP2), Magnetic, and Provisioned IOPS (SSD). You can choose different types of volumes depending upon the nature of your job. Our internal testing suggests that the General Purpose SSD volumes should suffice for most of the workloads, however we recommend that you test against your own workload. One thing to note is that the General Purpose SSD volumes provide a baseline performance of 3 IOPS/GiB (up to 10,000 IOPS) with the ability to burst to 3,000 IOPS for volumes under 1,000 GiB. Please see I/O Credits and Burst Performance for more details. Here is a comparison of the volumes types:

General Purpose (SSD) Provisioned IOPS Magnetic
Storage Media  SSD-backed  SSD-backed  Magnetic-backed
Max Volume Size  16 TB  16 TB  1 TB
Max IOPS per Volume  10,000 IOPS  20,000 IOPS  ~100 IOPS
Max IOPS Burst Performance  3000 IOPS for volumes <= 1TB  n/a  Hundreds
Max Throughput per Volume 160 MB/second  320 MB/second  40-90 MB/second
Max IOPS per Node (16K)   48,000  48,000  48,000
Max Throughput per Instance 800 MB/second  800 MB/second  800 MB/second
Latency (Random Read)   1-2 ms  1-2 ms  20-40 ms
API Name  gp2  io1  standard

Support for M4 and C4 Instances
You can now launch EMR clusters that use M4 and C4 instances in regions where they are available. The M4 instances feature a custom Intel Xeon E5-2676 v3 (Haswell) processor and the C4 instances are based on the Intel Xeon E5-2666 v3 processor. These instances are designed to deliver the highest level of processor performance on EC2. Both types of instances offer Enhanced Networking which delivers up to 4 times the packet rate of instances without Enhanced Networking, while ensuring consistent latency, even when under high network I/O. Both the M4 and C4 instances are EBS-Optimized by default, with additional, dedicated network capacity for I/O operations. The instances support 64-bit HVM AMIs and can be launched only within a VPC.

Please see the Amazon EMR Pricing page for more details on the prices for these instances.

Productivity Tip
You can generate a create-cluster command that represents the configuration of an existing EMR 4.x cluster, including the EBS volumes. This will allow you to recreate the cluster using the AWS Command Line Interface (CLI).

Available Now
These new features are available now and you can start using them today!

Abhishek Sinha, Senior Product Manager

AWS Webinars for February & Early March 2016

by Jeff Barr | on | in Webinars | | Comments

Last month I challenged you to learn something new in 2016 and offered up our January webinar series as a starting point. We’ve put together another strong lineup for February and I’d like to share it with you now. As always, the webinars are free but space is limited and you should sign up ahead of time if you would like to attend.

Here’s what we have in store for February and early March (all times are Pacific):

Tuesday, February 23
Many developers are using containers to simplify the packaging, deployment, and operation of their applications. This deep-dive webinar will show you how to use Amazon EC2 Container Service to simplify the use of Docker in production.

The AWS Database Migration Service will help you to migrate your existing relational data to the cloud. This webinar will tell you how:

Do you want to allow your on-premises applications to take advantage of cloud storage? Learn more about some hybrid storage models in this webinar:

Wednesday, February 24
Are you eager to get business value from your Big Data projects? Attend this webinar to learn how your peers and competitors are doing it!

Are you puzzled by the term NoSQL? Learn about what it is and what it can do for you in this webinar:

Building mobile apps? Learn how to use AWS Device Farm to test them!

Thursday, February 25
New environments bring new security challenges! If you are building IoT (Internet of Things) applications, this webinar is a must!

AWS Lambda gives you the power to build serverless applications that require no administration and can scale with no effort. Join our webinar to learn how Lambda can now access functionality within a VPC (Virtual Private Cloud):

Handling Big Data requires some architectural considerations and design patterns that might not be obvious at first. Learn more at this webinar:

Tuesday, March 1 & Thursday, March 3
Migrating your Windows-based applications to the cloud does not necessarily entail stepping out of a familiar and comfortable environment. During our virtual hands-on labs an instructor will guide you through the process of configuring, launching, and securing a Windows virtual machine in the AWS Cloud:

Jeff;

PS – Your suggestions for future webinars are welcome! Leave me a comment and I will share it with the team.

 

 

AWS Week in Review – February 8, 2016

by Jeff Barr | on | in Week in Review | | Comments

Let’s take a quick look at what happened in AWS-land last week:

Monday

February 8

Tuesday

February 9

Wednesday

February 10

Thursday

February 11

Friday

February 12

Saturday

February 13

Sunday

February 14

New & Notable Open Source

  • Vault is a tool for managing secrets.
  • credstash is a little utility for managing credentials in the cloud.
  • microservices-playground is for microservices running on AWS in a Docker Container using ECS.
  • aws-role-editor is a Google Chrome extension to modify roles in the AWS Console.
  • aws-request-signer is a Google Chrome extension that signs requests to AWS endpoints using SigV4.
  • conan-aws-lambda is an AWS Lambda plugin for Conan the Deployer.
  • aws-nats is a Python and CloudFormation script to run a NATS cluster in AWS.
  • shepherd is a framework for building APIs using AWS API Gateway and AWS Lambda.
  • aws-lambda-ffmpeg ia an AWS Lambda function that resizes a video and outputs a thumbnail using FFmpeg.
  • aerosol is a DSL and Gem for defining an AWS architecture.

New SlideShare Presentations

New Customer Success Stories

  • Bazaarvoice –  The company offers a technology platform and services that help customers collect and analyze consumer content, using that data to increase sales and improve their products and services.
  • Bitdefender – By using AWS, the developers at Bitdefender have additional tools to innovate and scale on demand with near-zero downtime, offering customers flexible, cost-effective security solutions.
  • The Globe and Mail – The Globe and Mail is using AWS to deliver dynamic, personalized content to its readers, helping boost reader engagement by 25 percent.
  • gumi Asia – Using AWS has enabled gumi Asia to achieve 99.5 percent availability, support demand peaks 50 percent higher than normal with no impact on performance, and avoid system redundancy and staffing costs.
  • Intermountain Healthcare – Intermountain Healthcare, using AWS and working with APN partner Syapse, can provide fast, cloud-based services to oncologists across the United States so they can deliver precision medicine to cancer patients.
  • Invision – Using AWS, InVision can offer its workforce management solution at one tenth of the cost of its physical software product, helping it to lower its costs to contact-center customers and reach a previously untapped 85 percent of the market.

New YouTube Videos

Upcoming Events

Help Wanted

Stay tuned for next week! In the meantime, follow me on Twitter and subscribe to the RSS feed.

Jeff;

Resources for Migrating Parse Applications to AWS

by Jeff Barr | on | in Amazon Mobile Hub | | Comments

In light of the recent announcement that Parse will be winding down, the AWS team has been working to provide developers with some migration paths and some alternative services, as have members of the AWS community. Here’s what I know about:

I also have some partner and community resources for you:

Migration Webinar
My colleague John Bury (Principal Solution Architect) has been working in the mobile space for more than 12 years. On March 1st he will lead a 200-level webinar, Migrating Mobile Apps from Parse to AWS. The webinar will run from 11 AM to Noon (PT). After an introductory look at the full range of AWS mobile services, John will lead you though the steps necessary to migrate your mobile app from Parse to AWS.

Jeff;

PS – Please share additional resources in the comment section and I’ll add them to the post.