AWS Official Blog

New – Notifications for AWS CodeDeploy Events

by Jeff Barr | on | in AWS CodeDeploy | | Comments

AWS CodeDeploy is a service that helps you to deploy your code to a fleet of EC2 or on-premises instances while taking care to leave as much of the fleet online as possible. CodeDeploy was designed to work with fleets that range in size from a single instance all the way up to thousands of instances (see my post, New AWS Tools for Code Management and Deployment for more information).

Notifications for CodeDeploy
In order to make it easier for you to use CodeDeploy as a part of your overall build, test, and deployment pipeline, we are introducing a new notification system today. You can now create triggers that send Amazon SNS notifications before, during, and after the deployment process for your applications. Triggers can be set for the deployment as a whole or for the individual instances targeted by the deployment, and are sent on both successes and failures. Here is the full list of triggers:

  • DEPLOYMENT_START
  • DEPLOYMENT_SUCCESS
  • DEPLOYMENT_FAILURE
  • DEPLOYMENT_STOP
  • INSTANCE_START
  • INSTANCE_SUCCESS
  • INSTANCE_FAILURE

You can create up to 10 triggers per application. You can connect several triggers to a single topic, or you can send each trigger to a distinct topic. You can use any of the delivery protocols supported by SNS (http, https, email, SMS, and mobile push). You can also invoke a Lambda function.

Creating a Trigger
Triggers can be created using the AWS Management Console or the AWS Command Line Interface (CLI). I’ll use the Console in the post. I set up the sample CodeDeploy application (three t2.micro instances):

And then I did an initial deployment:

Then I created an SNS topic, subscribed to it via email, and confirmed the subscription:

I returned to the Console and opened up my application within CodeDeploy:

I opened up IAM in another tab and updated the policy associated with the Service Role, giving it permission to write to SNS (you won’t need to do this if you select the managed AWSCodeDeployRole for your application):

Back in CodeDeploy, I clicked on Create trigger, entered a name, chose my events (all of them for this example), and selected my SNS topic from the dropdown:

I could also have chosen individual events:

My trigger was created and displayed in the Console:

CodeDeploy sent a confirming message to the topic and it was at the top of my Inbox:

Then I initiated a deployment and waited for the emails to arrive! Here’s what they looked like:

I used email for illustrative purposes; in a real-world application you would probably want to write some code to catch and handle the messages.

Triggers are available now and you can start using them today!

Jeff;

 

 

US Veterans Now Eligible For AWS Certification Exam Reimbursement

by Jeff Barr | on | in Training and Certification | | Comments

Service members returning from active duty can face many challenges—including finding a new job or even a new career. To help veterans succeed, we’ve worked with the Department of Veterans Affairs to make AWS Certification exams eligible for VA reimbursement under the GI Bill’s education provision.

About AWS Certifications
AWS Certifications recognize IT professionals with the technical expertise to design, deploy, and operate applications and infrastructure on AWS. Career transitions are never easy, but cloud IT presents one promising path forward—especially when industry surveys show that these skills remain in short supply.

Our hope is that easier access to our certification exam, combined with the unique talents veterans already possess, will open up more career possibilities for retired servicemen and servicewomen and help them achieve success in their post-military careers.

How it Works
Qualifying US veterans covered under a GI Bill with an education provision can now submit a reimbursement request to the Department of Veterans Affairs for exams taken after December 10, 2015 and purchased from Webassessor. The VA will cover exam fees up to $2000 (costs connected with preparing for a certification such as training courses or practice exams are not reimbursable).

Learn more about the benefit—or explore all AWS Training courses, videos, labs, and certifications.

Jeff;

 

Migrating to Amazon Aurora: The View from the Other Side

by Jeff Barr | on | in Amazon Aurora, Guest Post | | Comments

Today I have a new guest post, this one written by AWS customer Safe Software. They were an early adopter of Amazon Aurora and have a great story to tell.

Jeff;

Being in the business of moving data, Safe Software is always working to support leading edge database technology. This focus naturally led us to us being aware of Amazon Aurora.

In addition to leveraging the power of AWS and Aurora for our clients, we also evaluate new technologies from the perspective of improving our internal processes. When Aurora was released in beta, we immediately thought of migrating to it for our own systems. That decision has proven to be worthwhile. The move to Aurora has increased our productivity while providing an annual 40% cost reduction in systems costs.

Now that the migration is behind us, I’d like to share some tips and insights to those considering taking the leap. While I’d like to include the migration details, there is not much to say as it only took the click of a button. Instead I will share what we tried first, how we prepared, and how we optimized our systems further once they were operating in Aurora.

Why We Needed The Cloud
To ensure high quality of our spatial data transformation technology, FME, we run a grueling automated test suite. The platform supports 365+ data formats and limitless transformations, making the automated daily testing demanding: 15,000 tests x 4 operating systems x 3 products, running 24/7.

The problem: We couldn’t keep provisioning hardware to keep the system happy. Never mind keeping up with our expectation of a 1-second response time.

Our internal production database runs on a high traffic system with 140+ tables containing ~100 million rows of data. It is the primary operational repository for our build and test systems, as well as our internal intranet and reporting framework, supporting our server farms upwards of 150 machines. Over 100 users rely on this system.

What We Tried Before Aurora
We initially tried moving everything to MySQL on RDS, but found that we needed to run a read replica on a sizable instance to manage load. Even still, we were crowding ourselves against the productivity ceiling for the number of connections we could handle for most queries. We struggled to meet the needed response times. This solution had also immediately doubled our costs.

We’d spent so much time getting good at MySQL that the idea of having to relearn all of that in a new system was painful. Having something you treat like an appliance is much better.

Fail-Safe Preparations and Migration
We heard Aurora mirrors the MySQL experience, so we figured it was worth trying. To ensure we had nothing to lose we decided to keep the production system running in its existing form, while we tested Aurora.

One of the benefits of moving to a higher performance system is you have a good opportunity to re-assess a system that dates back years. During this migration we did some housekeeping. We looked at indexes, table structures, and many other relational aspects of the database. We were able to combining multiple schemas into just two for simpler logic.

The actual move into Aurora was trivial. Within the Amazon control panel, we chose to migrate – clicking a button. It happened in the background, and we ended up with a copy in Aurora of what we had running in MySQL! It was that simple!

Managing the cutover is a fairly big thing in terms of scheduling, to make sure we’re not impacting operations and meanwhile capturing a current snapshot of the data. We were wary that things could get out of sync; that by the time the migration was done the read dates may be out of date. We knew it would take a few hours to migrate the production system that was still operating, and during that time, data could change.

We chose to do the migration overnight on the live system while it was still running. We followed up with our own FME product to capture changes that had taken place in volatile tables during the migration (about 2-3% of our data), and port them over.

Our build and release team was able to manage the migration ourselves, and only involved the IT department to configure identity and access management and then change the DNS on our network once we’d verified that everything was a go.

We had checked a few examples for sanity first, but it was kind of a leap in the dark because we were early adopters. But we knew that we could just roll back to the old system if needed.

Optimizing The Experience Post-Migration
We thoroughly tested all of our processes afterward. There was some head-scratching after the first couple of days of monitoring; we experienced patches of heavy CPU load and slow-downs in Aurora during operations that had previously been unremarkable in MySQL.

We tracked these down to a set of inefficient queries using deeply nested SELECTs which were not readily modifiable. We resolved these issues with some simple schema changes, and pre-canning some of the more complex relationships using our own product, FME. Bottom Line: Schema design is still important.

No other issues were experienced during or since, and tuning these queries and indexes was ultimately beneficial. In operation we now have enterprise scale with the familiar interfaces we are used to.

For almost all operations, Aurora has proven faster, and it gives us more scalability. We’re running a relatively modest setup now, knowing that we can expand, and it’s saving us about $8,000 per year (60% cheaper). In fact, we could double our performance using Aurora and it would still be less than we paid last year. We save another $2,000 by reserving the instance for annual operations.

Operation management is pretty critical stuff, so it’s a relief not to worry about backups or database failures. The managed database saves us a lot of headaches. To achieve this performance ourselves would have a required huge investment in both hardware and personnel.

With Aurora, we can create our FME product builds better, faster, and the test results come through quickly, which ultimately means we can provide a higher quality product.

Iain McCarthy, Product Release Manager at Safe Software Inc.

Amazon EMR Update – Support for EBS Volumes, and M4 & C4 Instance Types

by Jeff Barr | on | in Amazon Elastic Block Store, Amazon EMR | | Comments

My colleague Abhishek Sinha wrote the guest post below to tell you about the latest additions to Amazon EMR.

Jeff;

Amazon EMR is a service that allows you to use distributed data processing frameworks such as Apache Hadoop, Apache Spark and Presto to process data on a managed cluster of EC2 instances.

Newer versions of EMR (3.10 and 4.x), allow you to use Amazon EBS volumes to increase the local storage of each instance. This works well with the existing set of supported instance types, and also gives you the ability to use the M4 and C4 instance types with EMR. Today I would like to tell you more about both of these features.

Increasing Instance Storage Using Amazon EBS
EMR uses the local storage of each instance for HDFS (Hadoop Distributed File System) and to store intermediate files when processing data from S3 using EMRFS. You can now use EBS volumes to extend this storage. The EBS volumes are tied to the lifecycle of the associated instances and augment any existing storage on the instance. If you terminate a cluster, any associated EBS volumes are also deleted along with it.

You will benefit from the ability to customize the storage of your EMR instances if…

  1. Your processing requirements demand a larger amount of HDFS (or local) storage than what is available by default on an instance. With support for EBS volumes, you will be able to customize the storage capacity on an instance relative to the compute capacity that the instance provides. Optimizing the storage on an instance will allow you to save costs.
  2. You want to take advantage of the latest generation EC2 instance types such as the M4, C4, and R3 and need more storage than is available on these instance types. You can now add EBS volumes to customize the storage in order to better meet your needs. If you’re using the older M1 and M2 instances, you should be able to reduce costs and improve performance by moving to  newer M4, C4 and R3 instances. We recommend that you benchmark your application to measure the impact on your specific workloads.

It’s important to note that the EBS volumes added to an Amazon EMR cluster do not persist data after the cluster is shutdown. EMR will automatically clean up the volumes when you terminate your cluster.

Adding EBS Volumes to a Cluster
EMR currently groups the nodes in your cluster into 3 logical instance groups: a Master Group, which runs the YARN Resource Manager and the HDFS Name Node Service; a Core Group, which runs the HDFS DataNode Daemon and the YARN Node Manager Service; and Task Groups, which run the YARN Node Manager Service. EMR supports up to 50 instance groups per cluster and allows you to select an instance type for each group. You can now specify the amount of EBS storage you want to add to each instance in a given instance group. You can specify multiple EBS volumes, add EBS volumes to instances with instance storage, or even combine different volumes of different types. Here is how you specify your storage configuration in the EMR Console:

For example, if you configured a Core Group to use the m4.2xlarge instance, attached a pair of 1 TB gp2 (General Purpose SSD) volumes and want 10 instances in the group, the Core group would have 10 instances with a total of 20 volumes. Here’s how you would set that up:

To learn more, read the EBS FAQ. Support for EBS is available starting with AMI 3.10 and EMR Release 4.0.

EBS Volume Performance Characteristics
Amazon EMR allows you to use several different EBS volume types: General Purpose SSD (GP2), Magnetic, and Provisioned IOPS (SSD). You can choose different types of volumes depending upon the nature of your job. Our internal testing suggests that the General Purpose SSD volumes should suffice for most of the workloads, however we recommend that you test against your own workload. One thing to note is that the General Purpose SSD volumes provide a baseline performance of 3 IOPS/GiB (up to 10,000 IOPS) with the ability to burst to 3,000 IOPS for volumes under 1,000 GiB. Please see I/O Credits and Burst Performance for more details. Here is a comparison of the volumes types:

General Purpose (SSD) Provisioned IOPS Magnetic
Storage Media  SSD-backed  SSD-backed  Magnetic-backed
Max Volume Size  16 TB  16 TB  1 TB
Max IOPS per Volume  10,000 IOPS  20,000 IOPS  ~100 IOPS
Max IOPS Burst Performance  3000 IOPS for volumes <= 1TB  n/a  Hundreds
Max Throughput per Volume 160 MB/second  320 MB/second  40-90 MB/second
Max IOPS per Node (16K)   48,000  48,000  48,000
Max Throughput per Instance 800 MB/second  800 MB/second  800 MB/second
Latency (Random Read)   1-2 ms  1-2 ms  20-40 ms
API Name  gp2  io1  standard

Support for M4 and C4 Instances
You can now launch EMR clusters that use M4 and C4 instances in regions where they are available. The M4 instances feature a custom Intel Xeon E5-2676 v3 (Haswell) processor and the C4 instances are based on the Intel Xeon E5-2666 v3 processor. These instances are designed to deliver the highest level of processor performance on EC2. Both types of instances offer Enhanced Networking which delivers up to 4 times the packet rate of instances without Enhanced Networking, while ensuring consistent latency, even when under high network I/O. Both the M4 and C4 instances are EBS-Optimized by default, with additional, dedicated network capacity for I/O operations. The instances support 64-bit HVM AMIs and can be launched only within a VPC.

Please see the Amazon EMR Pricing page for more details on the prices for these instances.

Productivity Tip
You can generate a create-cluster command that represents the configuration of an existing EMR 4.x cluster, including the EBS volumes. This will allow you to recreate the cluster using the AWS Command Line Interface (CLI).

Available Now
These new features are available now and you can start using them today!

Abhishek Sinha, Senior Product Manager

AWS Webinars for February & Early March 2016

by Jeff Barr | on | in Webinars | | Comments

Last month I challenged you to learn something new in 2016 and offered up our January webinar series as a starting point. We’ve put together another strong lineup for February and I’d like to share it with you now. As always, the webinars are free but space is limited and you should sign up ahead of time if you would like to attend.

Here’s what we have in store for February and early March (all times are Pacific):

Tuesday, February 23
Many developers are using containers to simplify the packaging, deployment, and operation of their applications. This deep-dive webinar will show you how to use Amazon EC2 Container Service to simplify the use of Docker in production.

The AWS Database Migration Service will help you to migrate your existing relational data to the cloud. This webinar will tell you how:

Do you want to allow your on-premises applications to take advantage of cloud storage? Learn more about some hybrid storage models in this webinar:

Wednesday, February 24
Are you eager to get business value from your Big Data projects? Attend this webinar to learn how your peers and competitors are doing it!

Are you puzzled by the term NoSQL? Learn about what it is and what it can do for you in this webinar:

Building mobile apps? Learn how to use AWS Device Farm to test them!

Thursday, February 25
New environments bring new security challenges! If you are building IoT (Internet of Things) applications, this webinar is a must!

AWS Lambda gives you the power to build serverless applications that require no administration and can scale with no effort. Join our webinar to learn how Lambda can now access functionality within a VPC (Virtual Private Cloud):

Handling Big Data requires some architectural considerations and design patterns that might not be obvious at first. Learn more at this webinar:

Tuesday, March 1 & Thursday, March 3
Migrating your Windows-based applications to the cloud does not necessarily entail stepping out of a familiar and comfortable environment. During our virtual hands-on labs an instructor will guide you through the process of configuring, launching, and securing a Windows virtual machine in the AWS Cloud:

Jeff;

PS – Your suggestions for future webinars are welcome! Leave me a comment and I will share it with the team.

 

 

AWS Week in Review – February 8, 2016

by Jeff Barr | on | in Week in Review | | Comments

Let’s take a quick look at what happened in AWS-land last week:

Monday

February 8

Tuesday

February 9

Wednesday

February 10

Thursday

February 11

Friday

February 12

Saturday

February 13

Sunday

February 14

New & Notable Open Source

  • Vault is a tool for managing secrets.
  • credstash is a little utility for managing credentials in the cloud.
  • microservices-playground is for microservices running on AWS in a Docker Container using ECS.
  • aws-role-editor is a Google Chrome extension to modify roles in the AWS Console.
  • aws-request-signer is a Google Chrome extension that signs requests to AWS endpoints using SigV4.
  • conan-aws-lambda is an AWS Lambda plugin for Conan the Deployer.
  • aws-nats is a Python and CloudFormation script to run a NATS cluster in AWS.
  • shepherd is a framework for building APIs using AWS API Gateway and AWS Lambda.
  • aws-lambda-ffmpeg ia an AWS Lambda function that resizes a video and outputs a thumbnail using FFmpeg.
  • aerosol is a DSL and Gem for defining an AWS architecture.

New SlideShare Presentations

New Customer Success Stories

  • Bazaarvoice –  The company offers a technology platform and services that help customers collect and analyze consumer content, using that data to increase sales and improve their products and services.
  • Bitdefender – By using AWS, the developers at Bitdefender have additional tools to innovate and scale on demand with near-zero downtime, offering customers flexible, cost-effective security solutions.
  • The Globe and Mail – The Globe and Mail is using AWS to deliver dynamic, personalized content to its readers, helping boost reader engagement by 25 percent.
  • gumi Asia – Using AWS has enabled gumi Asia to achieve 99.5 percent availability, support demand peaks 50 percent higher than normal with no impact on performance, and avoid system redundancy and staffing costs.
  • Intermountain Healthcare – Intermountain Healthcare, using AWS and working with APN partner Syapse, can provide fast, cloud-based services to oncologists across the United States so they can deliver precision medicine to cancer patients.
  • Invision – Using AWS, InVision can offer its workforce management solution at one tenth of the cost of its physical software product, helping it to lower its costs to contact-center customers and reach a previously untapped 85 percent of the market.

New YouTube Videos

Upcoming Events

Help Wanted

Stay tuned for next week! In the meantime, follow me on Twitter and subscribe to the RSS feed.

Jeff;

Resources for Migrating Parse Applications to AWS

by Jeff Barr | on | in Amazon Mobile Hub | | Comments

In light of the recent announcement that Parse will be winding down, the AWS team has been working to provide developers with some migration paths and some alternative services, as have members of the AWS community. Here’s what I know about:

I also have some partner and community resources for you:

Migration Webinar
My colleague John Bury (Principal Solution Architect) has been working in the mobile space for more than 12 years. On March 1st he will lead a 200-level webinar, Migrating Mobile Apps from Parse to AWS. The webinar will run from 11 AM to Noon (PT). After an introductory look at the full range of AWS mobile services, John will lead you though the steps necessary to migrate your mobile app from Parse to AWS.

Jeff;

PS – Please share additional resources in the comment section and I’ll add them to the post.

 

Amazon Web Services to Acquire NICE

by Jeff Barr | on | in Announcements | | Comments

I would like to extend a warm welcome to our new colleagues at NICE. We have signed an agreement to acquire this leading provider of software and services for high performance and technical computing.

Products for HPC
From their headquarters in Asti, Italy, NICE delivers products and solutions to customers all over the world. These products help customers to optimize and centralize their high performance computing (HPC) and visualization workloads while also providing tools that are a great fit for distributed workforces making use of mobile devices.

For Existing Customers
The NICE brand and team will remain intact and will continue to develop and support the EnginFrame and Desktop Cloud Visualization (DCV) products. Customers will continue to receive world-class support and services, enhanced with the backing of the AWS team. Going forward, NICE and AWS will work together to create even better tools and services.

Still Day 1
As Jeff Bezos often says, it is still day 1 and we don’t have all of the answers yet. However, I did want to share this news with you and let you know that we are looking forward to meeting and working with our new colleagues. We expect the deal to close in Q1 of 2016.

Jeff;

 

New – Access Resources in a VPC from Your Lambda Functions

by Jeff Barr | on | in AWS Lambda | | Comments

A few months ago I announced that you would soon be able to access resources in a VPC from your AWS Lambda functions. I am happy to announce that this much-wanted feature is now available and that you can start using it today!

Your Lambda functions can now access Amazon Redshift data warehouses, Amazon ElastiCache clusters, Amazon Relational Database Service (RDS) instances, and service endpoints that are accessible only from within a particular VPC. In order to do this, you simply select one of your VPCs and identify the relevant subnets and security groups. Lambda uses this information to set up elastic network interfaces (ENIs) and private IP addresses (drawn from the subnet or subnets that you specified) so that your Lambda function has access to resources in the VPC.

Accessing Resources in a VPC
You can set this up when you create a new function. You can also update an existing function so that it has VPC access.  You can configure this feature from the Lambda Console or from the CLI.  Here’s how you set it up from the Console:

That’s all you need to do! Be sure to read Configuring a Lambda Function to Access Resources in an Amazon VPC in the Lambda documentation if you have any questions.

Things to Know
Here are a couple of things that you should know about this new feature:

ENI & IP Address Resources – Because Lambda automatically scales based on the number of events that is needs to process, your VPC must have an adequate supply of free IP addresses on the designated subnets.

Internet Access – As soon as you enable this functionality for a particular function, the function no longer has access to the Internet by default. If your function requires this type of access, you will need to set up a Managed NAT Gateway in your VPC (see New – Managed NAT (Network Address Translation) Gateway for AWS for more information) or run your own NAT (see NAT Instances).

Security Groups – The security groups that you choose for a function will control the function’s access to the resources in the subnets and on the Internet.

S3 Endpoints – You can also use this feature to access S3 endpoints within a VPC (consult New – VPC Endpoint for Amazon S3 to learn more).

Webinar – To learn more about this new feature, join our upcoming webinar, Essentials: Introducing AWS VPC Support for AWS Lambda.

Jeff;

Amazon RDS Update – Share Encrypted Snapshots, Encrypt Existing Instances

by Jeff Barr | on | in Amazon RDS, Security | | Comments

We want to make it as easy as possible for you to secure your AWS environment. Some of our more recent announcements in this area include encrypted EBS boot volumes, encryption at rest for Amazon Aurora, and support for AWS Key Management Service (KMS) across several different services.

Today we are giving you some additional options for data stored in Amazon Relational Database Service (RDS). You can now share encrypted database snapshots with other AWS accounts. You can also add encryption to a previously unencrypted database instance.

Sharing Encrypted Snapshots
When you are using encryption at rest for a database instance, automatic and manual database snapshots of the instance are also encrypted. Up until now, encrypted snapshots were private to a single AWS account and could not be shared. Today we are giving you the ability to share encrypted snapshots with up to 20 other AWS accounts. You can do this from the AWS Management Console, AWS Command Line Interface (CLI), or via the RDS API. You can share encrypted snapshots within an AWS region, but you cannot share them publicly. As is the case with the existing sharing feature, today’s release applies to manual snapshots.

To share an encrypted snapshot, select it and click on Share Snapshot. This will open up the Manage Snapshot Permissions page. Enter one or more account IDs (click on Add after each one) and click on Save when you have entered them all:

The accounts could be owned by your organization (perhaps you have separate accounts for dev, test, staging, and production) or by your business partners. Backing up to your mission-critical databases to a separate AWS account is a best practice, and one that you can implement using this new feature while also gaining the benefit of encryption at rest.

After you click on Save, the other accounts have access to the shared snapshots. The easiest way to locate them is to visit the RDS Console and filter the list using Shared with Me:

The snapshot can be used to create a new RDS database instance. To learn more, read about Sharing a Database Snapshot.

Adding Encryption to Existing Database Instances
You can now add encryption at rest using KMS keys to a previously unencrypted database instance. This is a simple, multi-step process:

  1. Create a snapshot of the unencrypted database instance.
  2. Copy the snapshot to a new, encrypted snapshot. Enable encryption and specify the desired KMS key as you do so:
  3. Restore the encrypted snapshot to a new database instance:
  4. Update your application to refer to the endpoint of the new database instance:

And that’s all you need to do! You can use a similar procedure to change encryption keys for existing database instances. To learn more, read about Copying a Database Snapshot.

Jeff;