AWS Blog

Way back in 2010, we launched Resource Tagging for EC2 instances and other EC2 resources. Since that launch, we have raised the allowable number of tags per resource from 10 to 50, and we have made tags more useful with the introduction of resource groups and a tag editor. Our customers use tags to track ownership, drive their cost accounting processes, implement compliance protocols, and to control access to resources via IAM policies.

The AWS tagging model provides separate functions for resource creation and resource tagging. While this is flexible and has worked well for many of our users, it does result in a small time window where the resources exist in an untagged state. Using two separate functions means that resource creation could succeed only for tagging to fail, again leaving resources in an untagged state.

Today we are making tagging more flexible and more useful, with four new features:

Tag on Creation – You can now specify tags for EC2 instances and EBS volumes as part of the API call that creates the resources.

Enforced Tag Usage – You can now write IAM policies that mandate the use of specific tags on EC2 instances or EBS volumes.

Resource-Level Permissions – By popular request, the CreateTags and DeleteTags functions now support IAM’s resource-level permissions.

Enforced Volume Encryption – You can now write IAM policies that mandate the use of encryption for newly created EBS volumes.

Tag on Creation
You now have the ability to specify tags for EC2 instances and EBS volumes as part of the API call that creates the resources (if the call creates both instances and volumes, you can specify distinct tags for the instance and for each volume). The resource creation and the tagging are performed atomically; both must succeed in order for the operation (RunInstances, CreateVolume, and other functions that create resources) to succeed. You no longer need to build tagging scripts that run after instances or volumes have been created.

Here’s how you specify tags when you launch an EC2 instance (the CostCenter and SaveSnapshotFlag tags are also set on any EBS volumes created when the instance is launched):

To learn more, read Using Tags.

Resource-Level Permissions
CreateTags and DeleteTags now support IAM’s resource-level permissions, as requested by many customers. This gives you additional control over the tag keys and values on existing resources.

Also, RunInstances and CreateVolume now support additional resource-level permissions. This allows you to exercise control over the users and groups that can tag resources on creation.

To learn more, see Example Policies for Working with the AWS CLI or an AWS SDK.

Enforced Tag Usage
You can now write IAM policies that enforce the use of specific tags. For example, you could write a policy that blocks the deletion of tags named Owner or Account. Or, you could write a “Deny” policy that disallows the creation of new tags for specific existing resources. You could also use an IAM policy to enforce the use of Department and CostCenter tags to help you achieve more accurate cost allocation reporting. In order to implement stronger compliance and security policies, you could also restrict access to DeleteTags if the resource is not tagged with the user’s name. The ability to enforce tag usage gives you precise control over access to resources, ownership, and cost allocation.

Here’s a statement that requires the use of costcenter and stack tags (with values of “115” and “prod,” respectively) for all newly created volumes:

"Statement": [
    {
      "Sid": "AllowCreateTaggedVolumes",
      "Effect": "Allow",
      "Action": "ec2:CreateVolume",
      "Resource": "arn:aws:ec2:us-east-1:123456789012:volume/*",
      "Condition": {
        "StringEquals": {
          "aws:RequestTag/costcenter": "115",
          "aws:RequestTag/stack": "prod"
         },
         "ForAllValues:StringEquals": {
             "aws:TagKeys": ["costcenter","stack"]
         }
       }
     },
     {
       "Effect": "Allow",
       "Action": [
         "ec2:CreateTags"
       ],
       "Resource": "arn:aws:ec2:us-east-1:123456789012:volume/*",
       "Condition": {
         "StringEquals": {
             "ec2:CreateAction" : "CreateVolume"
        }
      }
    }
  ]

Enforced Volume Encryption
Using the additional IAM resource-level permissions now supported by RunInstances and CreateVolume, you can now write IAM policies that mandate the use of encryption for any EBS boot or data volumes created. You can use this to comply with regulatory requirements, enforce enterprise security policies, and to protect your data in compliance with applicable auditing requirements.

Here’s a sample statement that you can incorporate into an IAM policy for RunInstances and CreateVolume to enforce EBS volume encryption:

"Statement": [
        {
            "Effect": "Deny",
            "Action": [
                       "ec2:RunInstances",
                       "ec2:CreateVolume"
            ],
            "Resource": [
                "arn:aws:ec2:*:*:volume/*"
            ],
            "Condition": {
                "Bool": {
                    "ec2:Encrypted": "false"
                }
            }
        },

To learn more and to see some sample policies, take a look at Example Policies for Working with the AWS CLI or an AWS SDK and IAM Policies for Amazon EC2.

Available Now
As you can see, the combination of tagging and the new resource-level permissions on the resource creation and tag manipulation functions gives you the ability to track and control access to your EC2 resources.

This new feature is available now in all regions except AWS GovCloud (US) and China (Beijing). You can start using it today from the AWS Management Console, AWS Command Line Interface (CLI), AWS Tools for Windows PowerShell, or the AWS APIs.

We are planning to add support for additional EC2 resource types over time; stay tuned for more information!

— Jeff;

I’m in catch-up mode again, and would like to tell you about some recent improvements that we have made to Amazon Aurora. As a reminder, Aurora is our high-performance MySQL-compatible (and soon PostgreSQL-compatible) enterprise-class database (read Now Available – Amazon Aurora and Amazon Aurora – New Cost-Effective MySQL-Compatible Database Engine for Amazon RDS for an introduction).

Here’s are the newest additions to Aurora:

• Cross Region Snapshot Copy
• Cross Region Replication for Encrypted Databases
• Cross Account Encrypted Snapshot Sharing
• Availability in the US West (Northern California) Region
• T2.Small Instance Support

Let’s take a quick look at each one!

Cross Region Snapshot Copy
You can now copy Amazon Aurora snapshots (either automatic or manual) from one region to another. Select the snapshot, choose Copy Snapshot from the Snapshot Actions menu, pick a region, and enter a name for the new snapshot:

You can also choose to encrypt the snapshot as part of this operation. To learn more, read Copying a DB Snapshot or DB Cluster Snapshot.

Cross Region Replication for Encrypted Databases
You can already enable encryption when you create a fresh Amazon Aurora DB Instance:

You can now create a read replica in another region with a couple of clicks. You can use this to build multi-region, highly available systems or to move the data closer to the user. To create a cross region read replica, simply select the existing DB Instance and choose Create Cross Region Read Replica from the menu:

Then you choose the destination region in the Network & Security settings, and click on Create:

The destination region must include a DB Subnet Group that encompasses 2 or more Availability Zones.

To learn more about this powerful new feature, read Replicating Amazon Aurora DB Clusters Across AWS Regions.

Cross Account Encrypted Snapshot Sharing
You already have the ability to configure periodic, automated snapshots when you create your Amazon Aurora DB Instance. You can also create snapshots at any desired time with a couple of clicks:

If the DB Instance is encrypted, the snapshot will be as well.

You can now share encrypted snapshots with other AWS accounts. In order to use this feature, the DB Instance (and therefore the snapshot) must be encrypted with a Master Key other than the default RDS key. Select the snapshot and choose Share Snapshot from the Snapshot Actions menu:

Then enter the target AWS Account ID(s), clicking Add after each one, and click on Save to share the snapshot:

You will also need to share the key that was used to encrypt the snapshot. To learn more about this feature, read Sharing a DB Snapshot or DB Cluster Snapshot.

Availability in the US West (Northern California Region)
You can now launch Amazon Aurora DB Instances in the US West (Northern California) Region. Here’s the full list of regions where Aurora is available:

• US East (Northern Virginia)
• US East (Ohio)
• US West (Oregon)
• US West (Northern California)
• Canada (Central)
• EU (Ireland)
• EU (London)
• Asia Pacific (Tokyo)
• Asia Pacific (Sydney)
• Asia Pacific (Seoul)
• Asia Pacific (Mumbai)

See the Amazon Aurora Pricing page for pricing info in each region.

T2.Small Instance Support
You can now launch t2.small DB Instances:

These economical instances are a great fit for dev & test environments and for light production workloads. You can also use them to gain some experience with Amazon Aurora. These instances (along with six others, including the t2.medium that we launched last November) are available in all AWS regions where Aurora is available.

On-Demand pricing for t2.small DB Instances starts at $0.041 per hour in the US East (Northern Virginia) Region, dropping to $0.018 per hour for an All Upfront Reserved Instance with a 3 year term (see the Amazon Aurora Pricing page for more info).

— Jeff;

Customer service is essential to the success of any business. In order to provide voice-based customer service at scale, many organizations operate call centers. At the low end, call centers simply route incoming calls to any available agent. More sophisticated systems support more sophisticated routing and interaction, including the ability to create customized call trees and other Integrated Voice Response (IVR) systems. Traditionally, IVR systems have been difficult to install and expensive to license, with capacity-based pricing the norm.

The Amazon customer service organization aims to deliver an exceptional level of services to our customers. In order to do this at scale, the organization employs tens of thousands of agents working at more than 50 groups and subsidiaries centers scattered around the world, supporting customers that speak a wide variety of languages.

Welcome to Amazon Connect
Today I would like to introduce you to Amazon Connect. Building on the same technology used by many of our own customer service teams, Amazon Connect lets you set up a cloud-based contact center in minutes. You create your contact center, design your contact flows (similar to IVRs), and on-board your agents using a modern interface that is entirely web-based.

Instead of requiring help from IT teams and specialized consultants, Amazon Connect is simple enough to be configured and run directly by business decision makers. There’s no hardware to deploy and no per-agent licenses. Instead, you pay based on the number of customer-minutes and the amount of phone time that you consume. This scalable, pay-as-you-go model means that you can use Amazon Connect in situations where your call volume is unpredictable, spiky, or both.

Amazon Connect works with many different AWS services including:

Amazon S3 – Amazon Connect uses S3 to provide unlimited, encrypted storage of calls (audio) and reports.

AWS Lambda – This give you the ability to run code in serverless fashion as part of a customer contact. The code can pull data from your CRM or your database and use the data to provide a personalized customer experience.

Amazon Lex – Customer contacts can make use of natural language, conversational interfaces powered by the same technology behind Alexa. This aspect of Amazon Connect is being released in preview form and I’ll have more information about it some time soon.

AWS Directory Service – Amazon Connect can reference an existing Active Directory or it can create a new one. The directory is used to store user (administrator, manager, or agent) identities and permissions.

Amazon Kinesis – Amazon Connect can stream contact trace records (CTRs) to Amazon Kinesis. From there they can be pushed to Amazon S3 or Amazon Redshift and analyzed using Amazon QuickSight or other business analytics tools.

Amazon CloudWatch – Amazon Connect publishes real-time operational metrics to CloudWatch. These metrics will tell you how many calls are arriving per second, how many are being held in queues, and so forth. You can use these metrics to observe the performance of your contact center and to make sure that you have the right number of agents on hand.

Amazon Connect Benefits & Features
Let’s take a look at the major benefits and features of Amazon Connect.

Cloud-Powered – Amazon Connect is an integral part of AWS, and is designed to be robust, scalable, and highly available. Each contact center instance runs in multiple AWS Availability Zones.

Simple – As I already noted, Amazon Connect was designed to be set up and run by business decision makers. The graphical console makes the setup process (including the design of graphical call flows) easy and efficient.

Flexible – The Contact Flow Editor (CFE) that powers Amazon Connect includes blocks for interaction, integration, control flow, branching, and more. Call flows can include a mixture of prerecorded audio prompts, generated audio, Lex-powered interaction, integration with existing systems and databases, conversations with agents, call transfers, and more.

Economical – Amazon Connect‘s pay-as-you-go model keeps operating costs in line with actual usage. On the agent side, Amazon Connect includes a softphone that supports high quality audio.

Amazon Connect Tour
Let’s set up a customer contact center, starting at the Amazon Connect console. I click on Get started:

I can choose to use an existing directory or I can create a new one, which I’ll do. I’ll set up the URL for my contact center to proceed. This URL will allow my contact center’s users to sign in:

Next, I set myself up as the administrator:

My contact center can accept incoming calls, make outgoing calls, or both. Outgoing calls can be made as part of a contact flow or by an agent. I simply specify what I need (I’ll choose my direct dial and/or toll-free numbers later):

Next, I specify the S3 locations (buckets and prefixes) for call recording and reports. I can use separate buckets and prefixes for each, along with distinct AWS Key Management Service (KMS) encryption keys:

Then I confirm my choices and click on Create instance:

A few seconds later my Amazon Connect instance is ready to go, and I can click on Get started:

Amazon Connect logs me in using the administrator credentials that I entered earlier, and then I click on Let’s go to proceed:

Now I can get set up to accept my first call, by choosing a toll free or direct dial phone number:

I am ready to accept my first call:

I call the number on my cell and walk through the options. I press “1” to speak to an agent, and the softphone offers to let me (playing the role of the agent) take the call:

Now that the Amazon Connect is set up and accepting calls, I can explore the Dashboard and configure my contact center:

I can see the phone numbers that are associated with the call center:

And I can claim additional numbers (up to 10 direct inward dialing and 10 toll free), choosing from numbers in a wide list of countries:

At the bottom of the screen I can choose the contact flow that is activated when the number is called:

Returning to the dashboard, the next step is to set the hours of operation. I can see the list of existing hours of operation, and I can create a new one:

I can have multiple hours of operation, each of which can be attached to a queue or referenced from a contact flow.

Now I can create queues to model the routing of contacts (incoming calls) to agents. Queues can represent priorities (different levels of regular and premium support), or agents with different business or language skills.

I also have the ability to create prompts. These are audio snippets that can be referenced from a contact flow. I can upload an existing WAV file or I can record one from within the browser:

Next, I can create a new contact flow or edit an existing one. A contact flow defines the customer experience after they make contact. Here is the sample flow that is set up by default:

I can find the desired block in the menu, drag it on to the canvas, and connect it in to the flow:

Then I click on the header of the block to set it up. I can use a prompt, or I can choose between static or dynamic text to speech (dynamic speech would reference a value attached to the call flow, perhaps resulting from a database lookup):

One of the most powerful and general blocks allows me to call a Lambda function during the contact flow. The function can pull data from a database, connect to a CRM, or implement any desired type of business logic. The function is provided with key-value pairs as input, and returns the same as output. The returned values are associated with the execution of the flow.

The next step is to create a routing profile, consisting of one or more queues, with associated priorities. The priorities allow agents to service multiple queues in the proper order (lower numbers are higher priority); you can combine priorities and delays to set up routing rules that are in accord with the needs of your users and your contact center:

The final step is to add and empower the users (administrators, managers, agents, and so forth). This can be done on a user-by-user basis or by importing a CSV file that contains definitions for multiple users. Here’s how I add a user named Hello World:

I can also create a hierarchy of agents (great for regional and/or divisional reporting).

Amazon Connect lets me use security profiles to control the permissions assigned to each type of user:

I also have access to operational metrics, both real-time and historical. Here are some of the real-time metrics (the time frame and displayed fields can be customized as needed):

The metrics are published to CloudWatch, where they can be used to drive alarms.

I hope that this brief tour has given you a reasonably thorough introduction to Amazon Connect, and would love to hear how you have used it to modernize your contact center and to improve the level of service that you can provide to your customers:

As you can see, it is a rich and powerful service, with plenty of room for customization and extension. There are plenty of advanced features and configuration options that you can explore on your own.

Available Now
Amazon Connect is available now in the US East (Northern Virginia) Region and you can create a contact center today!

Pricing, as I noted, is on a pay-as-you-go basis. You pay for each contact center and each active phone call (inbound or outbound) on a per-minute basis. As part of the AWS Free Tier, you get 90 minutes of contact center usage, two phone numbers (DID and toll-free), 30 minutes of inbound DID calls, 30 minutes of inbound toll-free calls, and 30 minutes of outbound calls, all per month, for an entire year.

— Jeff;

PS – Visit the Amazon Connect Images album to see my entire LEGO build!

Last year I announced that AWS had signed an agreement to acquire NICE, and that we planned to work together to create even better tools and services for high performance and scientific computing.

Today I am happy to be able to tell you about the launch of NICE EnginFrame 2017. This product is designed to simplify the process of setting up and running technical and scientific applications that take advantage of the power, scale, and flexibility of the AWS Cloud. You can set up a fully functional HPC cluster in less than an hour and then access it through a simple web-based user interface. If you are already familiar with and using EnginFrame, you can keep running it on-premises or make the move to the cloud.

AWS Inside
Your clusters (you can launch more than one if you’d like) reside within a Virtual Private Cloud (VPC) and are built using multiple AWS services and features including Amazon Elastic Compute Cloud (EC2) instances running the Amazon Linux AMI, Amazon Elastic File System for shared, NFS-style file storage, AWS Directory Service for user authentication, and Application Load Balancers for traffic management. These managed services allow you to focus on your workloads and your work. You don’t have to worry about system software upgrades, patches, scaling of processing or storage, or any of the other responsibilities that you’d have if you built and ran your own clusters.

EnginFrame is launched from a AWS CloudFormation template. The template is parameterized and self-contained, and helps to ensure that every cluster you launch will be configured in the same way. The template creates two separate CloudFormation stacks (collections of AWS resources) when you run it:

Main Stack – This stack hosts the shared, EFS-based storage for your cluster and an Application Load Balancer that routes incoming requests to the Default Cluster Stack. The stack is also host to a set of AWS Lambda functions that take care of setting up and managing IAM Roles and SSL certificates.

Default Cluster Stack – This stack is managed by the Main Stack and is where the heavy lifting takes place. The cluster is powered by CfnCluster and scales up and down as needed, terminating compute nodes when they are no longer needed. It also runs the EnginFrame portal.

EnginFrame Portal
After you launch your cluster, you will interact with it using the web-based EnginFrame portal. The portal will give you access to your applications (both batch and interactive), your data, and your jobs. You (or your cluster administrator) can create templates for batch applications and associate actions for specific file types.

EnginFrame includes an interactive file manager and a spooler view that lets you track the output from your jobs. In this release, NICE added a new file uploader that allows you to upload several files at the same time. The file uploader can also reduce upload time by caching commonly used files.

Running EnginFrame
In order to learn more about EnginFrame and to see how it works, I started at the EnginFrame Quick Start on AWS page, selected the US East (Northern Virginia) Regions, and clicked on Agree and Continue:

After logging in to my AWS account, I am in the CloudFormation Console. The URL to the CloudFormation template is already filled in, so I click on Next to proceed:

Now I configure my stack. I give it a name, set up the network configuration, and enter a pair of passwords:

I finish by choosing an EC2 key pair (if I was a new EC2 user I would have to create and download it first), and setting up the configuration for my cluster. Then I click on Next:

I enter a tag (a key and a value) for tracking purposes, but leave the IAM Role and the Advanced options as-is, and click on Next once more:

On the next page, I review my settings (not shown), and acknowledge that CloudFormation will create some IAM resources on my behalf. Then I click on Create to get things started:

CloudFormation proceeds to create, configure, and connect all of the necessary AWS resources (this is a good time to walk your dog or say hello to your family; the process takes about half an hour):

When the status of the EnginFrame cluster becomes CREATE_COMPLETE, I can click on it, and then open up the Outputs section in order to locate the EnginFrameURL:

Because the URL references an Application Load Balancer with a self-signed SSL certificate, I need to confirm my intent to visit the site:

EnginFrame is now running on the CloudFormation stack that I just launched. I log in with user name efadmin and the password that I set when I created the stack:

From here I can create a service. I’ll start simple, with a service that simply compresses an uploaded file. I click on Admin’s Portal in the blue title bar, until I get to here:

Then I click on Manage, Services, and New to define my service:

I click on Submit, choose the Job Script tab, add one line to the end of the default script, and Close the action window:

Then I Save the new service and click on Test Run in order to verify that it works as desired. I upload a file from my desktop and click on Submit to launch the job:

The job is then queued for execution on my cluster:

This just scratches the surface of what EnginFrame can do, but it is all that I have time for today.

Availability and Pricing
EnginFrame 2017 is available now and you can start using it today. You pay for the AWS resources that you use (EC2 instances, EFS storage, and so forth) and can use EnginFrame at no charge during the initial 90 day evaluation period. After that, EnginFrame is available under a license that is based on the number of concurrent users.

— Jeff;

My colleague Gene Farrell introduced you to Amazon AppStream 2.0 late last year. In his guest post, Gene explained how AppStream 2.0 lets you run desktop applications securely on any device, from within the comfort of an HTML5 web browser (read the entire post to learn more). For example, I used the AppStream 2.0 Try it Now page to launch and then immediately start using Siemens Solid Edge. I simply chose the desired application from the Try it Now page:

I was running Solid Edge a few seconds later, no installation or setup needed:

By Popular Request – New Features for Enterprises, SMBs, and ISVs
Since that re:Invent launch, we have been fine-tuning AppStream 2.0, adding in some features that our customers have been asking for. These features will allow our customers to more easily deploy, access, manage and track the applications that they make available for use through AppStream. We’ve rolled most of these out without individual blog posts, and today I’d like to let you know what we’ve been up to. Here are the newest features:

Fleet Auto Scaling – This brand-new feature allows you to use the CloudWatch metrics to scale your fleet up and down in response to changes in demand. This allows you to deliver applications as economically as possible, while still providing instant access.

Image Builder – You can build your own AppStream 2.0 images that contain your choice of applications.

SAML 2.0 Authentication – You can use your existing SAML 2.0 compliant directory with AppStream 2.0. Your users can use their existing credentials to log in.

Fleet Management – You have additional management options for the instances that run your applications.

CloudWatch Metrics – You can observe and monitor seven Amazon CloudWatch metrics, including the size and overall utilization of your fleets.

Let’s take a look at each one!

Fleet Auto Scaling
This feature is brand new, and is powered by the new CloudWatch metrics! You can now associate scaling policies with each of your fleets and use them to meet varying levels of user demand and to control costs. If you are using AppStream 2.0 to deliver productivity applications to your users, you can use the scaling policies to ensure that capacity comes online as needed during office hours, and goes away in the evening when your users are done for the day. Here is a fleet with scale out (add capacity) and scale in (remove capacity) policies:

In order to take advantage of this feature, you set the minimum and maximum capacity when you create the fleet:

This will create the default policies, which you can later edit, add, or remove (you can have up to 50 policies per fleet). To learn more, read about AppStream Fleet Auto Scaling.

Image Builder
This feature allows you to create custom images that contain your choice of commercial or proprietary applications. In order to do this, you launch an instance called an image builder. Then you log in to the instance, install and configure the applications as desired, and capture the state of the instance as an image. The entire login and customization process takes place within your web browser; you don’t have to download any keys or remember any passwords. The application appears in the Image Registry and is available to your users.

I can launch an image builder from the AppStream 2.0 Console:

Next, I choose the starting point (an existing image):

Then I configure the builder by giving it a name, choosing an instance size, and setting up the VPC:

I click on Review, confirm my settings, and then wait for the builder to launch:

Then I can connect to the image builder, set up the apps, and create an image. I have my choice of two identities when I connect, Admin and Test:

I select ImageBuildAdmin and (when prompted for a password), click on Log me in in the Admin Commands menu:

After logging in, I launch the Image Assistant app and use it to install and test my apps:

To learn more, read about Image Builders and follow the Using an AppStream 2.0 Image Builder Tutorial.

SAML 2.0 Authentication
This feature allows you to use any external identity provider that supports SAML 2.0 including Active Directory Federation Services, PingFederate Server, Okta, or Shibboleth:

After you follow the directions in Setting Up SAML, your users can log in to AppStream 2.0 using their existing identity and credentials. You can manage users and groups, control access to applications based on the identity or location of the user, and use Multi-Factor Authentication (MFA). To learn more, read Enabling Single Sign-on Access to AppStream 2.0 Using SAML 2.0. If you have already set up federated access to the AWS Management Console, much of what you already know will apply.

Fleet Management
This feature gives me additional control over my fleets (groups of instances that are running applications for users). I can see all of my fleets on a single screen:

I can select a fleet and then act on it:

Some properties of a fleet can be edited at any time. Others, including the VPC properties, can only be edited after the fleet has been stopped. To learn more, read about Stacks and Fleets.

CloudWatch Metrics
AppStream publishes eight metrics to CloudWatch for each fleet:

• RunningCapacity – Number of instances running.
• InUseCapacity – Number of instances in use.
• DesiredCapacity – Number of instances that are either running or pending.
• AvailableCapacity – Number of idle instances available for use.
• PendingCapacity – Number of instances being provisioned.
• CapacityUtilization – Percentage of fleet being used.
• InsufficientCapacityError – Number of sessions rejected due to lack of capacity.

You can see these metrics from within the AppStream 2.0 Console:

These metrics will help you to measure overall usage to to fine tune the size of your fleet. As is the case with every CloudWatch metric, you can generate alerts and raise alarms when a metric is outside of the desired range. You could also use AWS Lambda functions to make changes to your environment or to generate specialized notifications. To learn more, read about Monitoring Amazon AppStream 2.0 Resources.

Available Now
All of these features are available now and you can start using them today!

— Jeff;

Ok, time for another peanut butter and chocolate post! Let’s combine EC2 Run Command (New EC2 Run Command – Remote Instance Management at Scale) and CloudWatch Events (New CloudWatch Events – Track and Respond to Changes to Your AWS Resources) and see what we get.

EC2 Run Command is part of EC2 Systems Manager. It allows you to operate on collections of EC2 instances and on-premises servers reliably and at scale, in a controlled and selective fashion. You can run scripts, install software, collect metrics and log files, manage patches, and much more, on both Windows and Linux.

CloudWatch Events gives you the ability to track changes to AWS resources in near real-time. You get a stream of system events that you can easily route to one or more targets including AWS Lambda functions, Amazon Kinesis streams, Amazon SNS topics, and built-in EC2 and EBS targets.

Better Together
Today we are bringing these two services together. You can now create CloudWatch Events rules that use EC2 Run Command to perform actions on EC2 instances or on-premises servers. This opens the door to all sorts of interesting ideas; here are a few that I came up with:

Final Log Collection – Collect application or system logs from instances that are being shut down (either manually or as a result of a scale-in operation initiated by Auto Scaling).

Error Log Condition – Collect logs after an application crash or a security incident.

Instance Setup – After an instance has started, download & install applications, set parameters and configurations, and launch processes.

Configuration Updates – When a config file is changed in S3, install it on applicable instances (perhaps determined by tags). For example, you could install an updated Apache web server config file on a set of properly tagged instances, and then restart the server so that it picks up the changes. Or, update an instance-level firewall each time the AWS IP Address Ranges are updated.

EBS Snapshot Testing and Tracking – After a fresh snapshot has been created, mount it on a test instance, check the filesystem for errors, and then index the files in the snapshot.

Instance Coordination – Every time an instance is launched or terminated, inform the others so that they can update internal tracking information or rebalance their workloads.

I’m sure that you have some more interesting ideas; please feel free to share them in the comments.

Time for Action!
Let’s set this up. Suppose I want to run a specific Linux shell command every time Auto Scaling adds another instance to an Auto Scaling Group.

I start by opening the CloudWatch Events Console and clicking on Create rule:

I configure my Event Source to be my Auto Scaling Group (AS-Main-1), and indicate that I want to take action when EC2 instances are launched successfully:

Then I set up the target. I choose SSM Run Command, pick the AWS-RunShellScript document, and indicate that I want the command to be run on the instances that are tagged as coming from my Auto Scaling group:

Then I click on Configure details, give my rule a name and a description, and click on Create rule:

With everything set up, the command service httpd start will be run on each instance launched as a result of a scale out operation.

Available Now
This new feature is available now and you can start using it today.

— Jeff;

Today we’re excited to introduce a new feature for Amazon EMR clusters called instance fleets. Instance fleets gives you a wider variety of options and intelligence around instance provisioning. You can now provide a list of up to 5 instance types with corresponding weighted capacities and spot bid prices (including spot blocks)! EMR will automatically provision On-Demand and Spot capacity across these instance types when creating your cluster. This can make it easier and more cost effective to quickly obtain and maintain your desired capacity for your clusters.

You can also specify a list of Availability Zones and EMR will optimally launch your cluster in one of the AZs. EMR will also continue to rebalance your cluster in the case of Spot instance interruptions by replacing instances with any of the available types in your fleet. This will make it easier to maintain your cluster’s overall capacity. Instance fleets can be used instead of instance groups. Just like groups your cluster will have master, core, and task fleets.

Let’s take a look at the console updates to get an idea of how these fleets work.

We’ll start by navigating to the EMR console and clicking the Create Cluster button. That should bring us to our familiar EMR provisioning console where we can navigate to the advanced options near the top left.

We’ll select the latest EMR version (instance fleets are available for EMR versions 4.8.0 and greater, with the exception of 5.0.x) and click next.

Now we get to the good stuff! We’ll select the new instance fleet option in the hardware options.

Now what I want to do is modify our core group to have a couple of instance types that will satisfy the needs of my cluster.

EMR will provision capacity in each instance fleet and availability zone to meet my requirements in the most cost effective way possible. The EMR console provides an easy mapping of vCPU to weighted capacity for each instance type, making it easy to use vCPU as the capacity unit (I want 16 total vCPUs in my core fleet). If the vCPU units don’t match my criteria for weighting instance types I can change the “Target capacity” selector to include arbitrary units and define my own weights (this is how the API/CLI consume capacity units as well).

When the cluster is being provisioned if it’s unable to obtain the desired spot capacity within a user defined timeout you can have it terminate or fall back onto On-Demand instances to provision the rest of the capacity.

All this functionality for instance fleets is also available from the AWS SDKs and the CLI. Let’s take a look at how we would provision our own instance fleet.

First we’ll create our configuration json in my-fleet-config.json:

[
  {
    "Name": "MasterFleet",
    "InstanceFleetType": "MASTER",
    "TargetOnDemandCapacity": 1,
    "InstanceTypeConfigs": [{"InstanceType": "m3.xlarge"}]
  },
  {
    "Name": "CoreFleet",
    "InstanceFleetType": "CORE",
    "TargetSpotCapacity": 11,
    "TargetOnDemandCapacity": 11,
    "LaunchSpecifications": {
      "SpotSpecification": {
        "TimeoutDurationMinutes": 20,
        "TimeoutAction": "SWITCH_TO_ON_DEMAND"
      }
    },
    "InstanceTypeConfigs": [
      {
        "InstanceType": "r4.xlarge",
        "BidPriceAsPercentageOfOnDemandPrice": 50,
        "WeightedCapacity": 1
      },
      {
        "InstanceType": "r4.2xlarge",
        "BidPriceAsPercentageOfOnDemandPrice": 50,
        "WeightedCapacity": 2
      },
      {
        "InstanceType": "r4.4xlarge",
        "BidPriceAsPercentageOfOnDemandPrice": 50,
        "WeightedCapacity": 4
      }
    ]
  }
]

Now that we have our configuration we can use the AWS CLI’s ’emr’ subcommand to create a new cluster with that configuration:

aws emr create-cluster --release-label emr-5.4.0 \
--applications Name=Spark,Name=Hive,Name=Zeppelin \
--service-role EMR_DefaultRole \
--ec2-attributes InstanceProfile="EMR_EC2_DefaultRole,SubnetIds=[subnet-1143da3c,subnet-2e27c012]" \
--instance-fleets file://my-fleet-config.json

If you’re eager to get started the feature is available now at no additional cost and you can find detailed documentation to help you get started here.

Thanks to the EMR service team for their help writing this post!

– Randall Hunt

Most of us equate in-memory caching with improved performance and lower cost at scale when designing applications or building solutions. Now if there was only a service that would continually make it simpler to deploy and utilize in-memory cache in the cloud while increasing the ability to scale.

Okay no more joking around, the cloud service that provides this great functionality is, of course, Amazon ElastiCache. Amazon ElastiCache is an AWS managed service that provides a performant in-memory data store or cache in the cloud while offering a straightforward way to create, scale, and manage a distributed environment for low-latency, secure, access of I/O intensive or compute heavy data. Additionally, ElastiCache reduces the overhead of managing infrastructure for your in-memory data structure server or cache by detecting and replacing failed nodes while providing enhanced visibility into key performance metrics of the caching system nodes via Amazon CloudWatch. This exciting service is now launching support for Enhanced Redis Backup and Restore with Cluster Resizing.

For those of you familiar with Amazon ElastiCache, you are likely aware that ElastiCache currently supports two in-memory key-value engines:

• Memcached: an open source, high-performing, distributed memory object caching system developed in 2003 with the initial goal of speeding up dynamic web applications by alleviating database load
• Redis: an open source in-memory data structure store launched in 2009 developed as a broker for caching, messaging, and databases with built-in replication, atomic operation support, various levels of on-disk persistence, and high availability via Redis Cluster.

In October of 2016, support was added for Redis Cluster with Redis 3.2.4. This allowed ElastiCache Redis users to, not only take advantage of Redis clusters, but also gave users the ability to:

• Create cluster-level backups.
• Produce snapshots of each of the cluster’s shards contained within backups.
• Scale their workloads with 3.5TiB of data across up to 15 shards.

You can read more about using Redis with ElastiCache and the related features by reviewing the product page for Amazon ElastiCache for Redis.

With the launch of the Enhanced Backup and Restore with Cluster Resizing feature, ElastiCache is providing even deeper support for Redis with a clear-cut migration path to a managed Redis Cluster experience. There are several benefits of this enhancement for ElastiCache and Redis users alike, such as:

• Ability to restore backup into a Redis Cluster with a different number of shards and slot distribution
• Deliver the capability for users to resize Redis workloads
• Allow Redis database file (RDB) snapshots as input for creating a sharded Redis Cluster
• Offer option to use snapshot(s) of Redis on EC2 implementations (both Redis Cluster and single-shard Redis) as data input for sharded Redis Cluster creation

To accomplish these tasks, ElastiCache will parse the Redis key space across the backup’s individual snapshots, and redistribute the keys in the new Cluster according to the requested number of shards and hash slots. You would simply take your RDB snapshots and store them on S3, then provide ElastiCache with the desired number of shards and the snapshot file. ElastiCache handles the heavy lifting of restoring the Redis data store into a Redis cluster.

I am sure that you all may be thinking; Is it really that easy to leverage the Enhanced Redis Backup and Restore with Cluster Resizing feature in ElastiCache? Well, there is no time like the present to find out. Let’s take a trip to the AWS Management Console, and put this newly launched enhancement in action by restoring an external RDB snapshot to a new cluster using ElastiCache.

My first stop in the AWS Management console is to the Amazon S3 console. I have some Redis .rdb snapshot files I received from some of my peers here at AWS in order to test the restore of an external Redis snapshot to ElastiCache. I will need to put these files into Amazon S3 so that I can access the snapshots as input for my ElastiCache Redis cluster.

In the S3 console, I will go to my S3 bucket, aws-blog-tew-posts, that I created for testing and development purposes. I’ll upload the .rdb snapshot files that were provided to me into this S3 bucket.

It is important to note that the name of your S3 bucket must conform to DNS standards. To be DNS-compliant, the name must be at least three characters, must contain only lowercase letters, numbers, and/or dashes, and it must start and end with a lowercase letter or number. While this may be obvious, I will also note that the bucket name cannot be in an IP address format. You can learn more about the S3 Bucket Restrictions at the link provided here.

With my .rdb files successfully uploaded into my aws-blog-tew-posts bucket, I need to take note of the S3 path to these backup files. For these files, the path would be aws-blog-tew-posts/dump_1.rdb or aws-blog-tew-posts/dump_10.rdb. If you have placed your files into a folder, the folder name would need to be included in this path, i.e. thebucketname/thefoldername/thefilename.

For ElastiCache to access these files, I need to ensure that the service has read permissions for each of the files. To provide access, I will update the permissions for each of .rdb files by assigning the Grantee as the canonical id for my region and grant the user Open/Download permissions. The canonical id for all regions, outside of China (Beijing) and AWS GovCloud (US), is as follows:

540804c33a284a299d2547575ce1010f2312ef3da9b3a053c8bc45bf233e4353

After I click the Save button, I am all set to use these files as input for an ElastiCache Redis cluster.

The next step is to go to the ElastiCache console. Here I will create a new ElastiCache Redis cluster and seed this new cluster with data from one of the RDB snapshots located in the files in my S3 bucket. I’ll choose the dump_1.rdb snapshot file to use as my data input to seed this new cluster. Since I want to explore the ElastiCache Redis capabilities added on this past October with 3.2.4 support of Redis Cluster, as well as, discuss the new Backup and Restore with Cluster Resizing enhancements, I’ll create a new Redis Cluster and ensure I have cluster mode enabled. At this point, I should note that you cannot restore from a backup created using a Redis (cluster mode enabled) cluster to a Redis (cluster mode disabled) cluster.

First, I will click the Get Started Now button from the ElastiCache console dashboard or the Create button based upon your console view.

In the Create your Amazon ElastiCache cluster dialog window, I’ll select Redis for my caching and make sure I click the checkbox for Cluster Mode enabled (Scale Out). The name of my new cluster will be, tew-rediscluster and I since I am enabling a Cluster mode, my ElastiCache Redis Engine version is 3.2.4. For this cluster, I will keep the default Redis port of 6379.

The key benefit of the ElastiCache enhanced Redis Backup and Restore feature is the cluster resizing capability that allows me to build a new cluster with a different number of shards than was originally used for the backup file. To build the new Redis Cluster, I am using only one RDB snapshot file, dump_1.rdb which is a small Redis instance backup with only one shard. However, in the creation of my new tew-rediscluster, I have opted for 3 shards with 2 replicas per shard.

In addition, I have the ability to specify a node type for my new cluster that is a different size than my original instance from the RDB snapshot. As I mentioned, the dump_1.rdb is a backup of a Redis instance that is significantly smaller than the size of the chosen node type for my tew-rediscluster shown below.

There are other options and data input needed in order to complete the creation of my ElastiCache Redis cluster that I will not show in this blog post. However, if you want to go through each of the steps necessary for creating an ElastiCache Redis cluster you can find more information in the AWS ElastiCache Getting Started documentation for Launch a Cluster.

Once I have provided all the information needed to create my ElastiCache Redis cluster, I will need to tell ElastiCache how to seed the cluster with the .rdb file by providing the file location from my S3 bucket. In the Import Data to Cluster section of the create dialog, I will enter the S3 path to my dump_1.rdb in the Seed RDB file S3 location textbox. Remember, the nomenclature for the S3 file path is Bucket/Folder/ObjectName so I will enter aws-blog-tew-posts/dump_1.rdb as the path to the RDB file in S3. All that is left now is to click the Create button.

That’s it! ElastiCache goes to work to creating the new Redis cluster. After a short time period, the ElastiCache console shows my new Amazon ElastiCache Redis cluster as available and I have successfully created this cluster with data restored from an external RDB snapshot file.

I just demonstrated how you have the capability to create an ElastiCache Redis cluster using an external RDB snapshot, but of course, you can create backups and restore from backups from your existing ElastiCache Redis clusters as well. To dig deeper into information about this newly launched feature, visit Restoring From a Backup with Cluster Resizing in the Amazon ElastiCache User Guide.

To learn more about making your applications more performant with Amazon ElastiCache, visit the AWS Amazon ElastiCache page for product details, resources, and customer testimonials.

– Tara

Today I would like to tell you about four S3 features that will give you detailed insights into your storage and your access patterns. You can see what and how much you are storing, how it is being used, and you can make informed decisions about the use of S3 storage classes as a result. These features will be of value to everyone who uses S3, whether they have tens, thousands, millions, or billions of objects in their buckets. Here’s an overview:

S3 Analytics – You can analyze the storage and retrieval patterns for your objects and use the results to choose the most appropriate storage class. You can inspect the results of the analysis from within the S3 Console, or you can load them into your favorite BI tool and dive deep. Either way, you now have the means to gain a deep understanding of your storage patterns and to see how they relate to usage and growth.

S3 Object Tagging – You can associate multiple key-value pairs (tags) with each of your S3 objects, with ability to change them at any time. The tags can be used to manage and control access, set up S3 Lifecycle policies, customize the S3 Analytics, and filter the CloudWatch metrics. You can think of the bucket as a data lake, and use tags to create a taxonomy of the objects within the lake. This is more flexible than using the bucket and a prefix, and allows you to make semantic-style changes without renaming, moving, or copying objects.

S3 Inventory – You can speed up your business workflows and your big data jobs using S3 Inventory. This feature provides you with a CSV-formatted flat-file representation of the contents of all or part (as identified by a prefix) of a bucket, on a daily or weekly basis.

S3 CloudWatch Metrics – You can improve the performance of your S3-powered applications by monitoring and alarming on 13 new Amazon CloudWatch metrics.

Let’s dive in…

S3 Analytics
As an S3 user, you have your choice of three storage classes (Standard, Standard – Infrequent Access, and Glacier) and the ability to use S3’s Object Lifecycle Management feature to indicate when objects should be either expired or transitioned to Standard – Infrequent Access or Glacier.

The S3 Analytics feature gives you the information that you need to have in order to choose between Standard and Standard – Infrequent Access for your objects. Because many customers store several different types or categories of information in a single bucket for ease of processing, you have the ability to run the analytics on subsets (defined by a common prefix or tag value) of the objects in a bucket. Each subset is defined by a filter; each bucket can have up to 1000 filters. Here are the filters on my jbarr-public bucket:

Here’s how I define a simple filter that analyzes objects that are prefixed with www:

I could choose to filter by tag name and value instead. Here’s how I would analyze objects that have a tag named type with the value page:

Each filter can have at most one prefix, along with any desired number of tags. I can also choose to have the analytics exported to a file each day:

Once one or more filters are in place, the analytics are run on a daily basis and I can view them by simply clicking on the filter. For example, here’s what I see when I click on my Archival filter:

There’s a lot of good information in this analysis! I can see that:

• Looking back 127 days, most of my objects that are older than 30 days are accessed infrequently. I can save money by creating a Lifecycle rule that will transition these objects to Standard – Infrequent Access storage.
• At this point, the majority (6.39 PB) of my storage is in Standard storage; just a tiny fraction (3.24 GB) is in Standard – Infrequent Access storage (I don’t actually have that much data in my bucket! The S3 team was kind enough to load up my account with some test metrics that were, shall we say, very generous).
• Over the 127 day observation period, between 16% and 30% of the Standard storage was retrieved on a per-day basis.
• Objects between 30-45 days old, 45-60 days old, 60-90 days old, 90-180 days old, and over 180 days old are all infrequently accessed and good candidates for Standard – Infrequent Access storage.

You can set up storage class analysis from the Console, the CLI, or through the S3 APIs.

To learn more, read S3 Storage Class Analysis.

S3 Object Tagging
Tags supplement the existing location-based S3 object management model (buckets and prefixes) with the ability to manage storage based on what the object represents. For example, you can tag objects with the name of a department and then construct IAM policies that grant access based on the tag. This gives you a lot of flexibility , including the ability to effect changes by simply altering tags.

You can create, update, or delete them during any part of the object’s life cycle. Tags can be referenced in IAM policies, S3 Lifecycle policies, and in the Storage Analytics filters that I just showed you. Each object can have up to ten tags and each version of an object has a distinct set of tags. You can use tags to manage the expiration of objects via a lifecycle policy and you can set up CloudWatch metrics that reflect the activity around a particular tag.

The Properties page for each object displays the current set of tags and allows me to edit them:

You can also set up and access tags from the CLI or through the S3 APIs. When you use either of these two methods, you must always work in terms of the full tag set. For example, if an object has four tags and you want to add a fifth one, you must read the current set, add the new one, and then update the entire set.

Tags can be replicated across regions via Cross-Region Replication, but your IAM policy on the source side must enable the s3:GetObjectVersionTagging and s3:ReplicateTags actions.

Tags cost $0.01 per 10,000 tags per month. Requests that add or update tags (PUT and GET, respectively) are charged at the usual rates.

To learn more, read about S3 Object Tagging.

S3 Inventory
S3’s LIST operation is synchronous, and returns up to 1000 objects at a time, along with a key that can be used to initiate a second LIST that picks up where the first one leaves off. While this works well for small to medium-sized buckets and single-threaded programs, it can be challenging to use in conjunction with huge buckets or multiple threads.

The S3 team told me that many of our customers store tens or hundreds of millions of objects in a single bucket, and that buckets with a billion or more objects are not unusual. These buckets are often processed daily or weekly as part of a larger workflow and can benefit from a more direct way to gain access to the list of objects in a bucket.

With S3 Inventory, you can now arrange to receive daily or weekly inventory reports for any of your buckets. You can use a prefix to filter the report and you can choose to include optional fields such as size, storage class, and replication status. Reports can be sent to an S3 bucket in your account or (with proper permission settings) in another account.

Here’s how I request a daily inventory report called WebStuff, for all objects that start with www:

I also need to give S3 permission to write to the destination bucket (jbarr-s3-inventory). Here’s the policy that I used (see Granting Permission for Amazon S3 Inventory and Amazon S3 Analytics to learn more):

The inventory mechanism creates three files: a manifest, a checksum for the manifest, and a data file. The manifest contains the location of the data file along with a checksum for it:

{
   "sourceBucket":"jbarr-public",
   "destinationBucket":"arn:aws:s3:::jbarr-s3-inventory",
   "version":"2016-11-30",
   "fileFormat":"CSV",
   "fileSchema":"Bucket, Key, Size, LastModifiedDate, ETag, StorageClass",
   "files":[
      {
         "key":"jbarr-public/DailyInventory/data/cf1da322-f52b-4c61-802a-b5c14f4f32e2.csv.gz",
         "size":2270,
         "MD5checksum":"be6d0eb3f9c4f497fe40658baa5a2e7c"
      }
   ]
}

Here’s what the data file looks like after it has been unzipped:

If you are using the inventory reports to power your daily or weekly processing workflow, don’t forget about S3 Notifications! The checksum file is written after the other two, so you can safely use it to get things moving. Also, don’t forget to use a lifecycle policy to manage your collection of inventory files since they will accumulate over time.

As an added bonus, using daily or weekly reports can save you up to 50%, when compared to multiple LIST operations. To learn more about this feature, read about S3 Storage Inventory.

S3 CloudWatch Metrics
S3 can now publish storage, request, and data transfer metrics to CloudWatch. The storage metrics are reported daily and are available at no extra cost. The request and data transfer metrics are available at one minute intervals (after some processing latency) and are billed at the standard CloudWatch rate. As is the case for the S3 Analytics, the CloudWatch metrics can be collected and reported for an entire bucket or for a subset selected via prefix or tags.

Here is the full set of metrics:

The metrics are available within the S3 and CloudWatch consoles. Here’s what the Total Request Latency looks like in the S3 Console for my bucket:

I can also click on View in CloudWatch and set CloudWatch alarms for any desired metric. Perhaps I want to be notified if my bucket grows beyond 100 MB:

To learn more, read How Do I Configure Request Metrics for an S3 Bucket?

Available Now
You have had access to these features since late last year!. They are accessible through the updated S3 Console, which also includes many other new features (watch Introduction to the New Amazon S3 Console to learn more).

— Jeff;

Calling all Game Developers! GDC 2017 was a blast in San Francisco a couple of weeks ago, so there is no better time to be inspired and passionate about learning and building cool games.

Therefore, I am excited to share that Amazon GameLift is now available for all C++ and C# game engines, including Amazon Lumberyard, Unreal Engine, and Unity, all with enhanced game session matching capabilities. For those of you not familiar with Amazon GameLift, let me introduce this managed service designed to aid game developers in delivering fun and innovative online game experiences.

Amazon GameLift is a managed AWS service for hosting dedicated game servers, making it easier for game developers to scale their game capacity and match players into available game sessions. With Amazon GameLift, you can host servers, track game availability, defend game servers from distributed denial of service (DDoS) attacks, and deploy updates without taking your game offline. The Amazon GameLift service powers dedicated game servers for Amazon Game Studios, as well as external game development customers, and is designed to support session-based games with game loops that start and end within a specified time.

The latest Amazon GameLift release enhances the current functionality of the service, as well as adding awesome new features to help simplify game development and deployment for developers. Let us review some of the cool features of the Amazon GameLift service:

• Multi-engine support: Initially, Amazon GameLift service could only be used with the Amazon Lumberyard game engine. The service is now enhanced to integrate with popular game engines like Unreal Engine, Unity, as well as, custom C# and C++ game engines.
• New server SDK language support: In order to support a larger set of customers and developers, the service provides an Amazon GameLift Server SDK available for C# and C++. This includes an Unreal Engine plugin, which is a customized version of the C++ Server SDK that is compatible with the Unreal Engine API for Amazon GameLift.
• Client SDK language support expansion: The Amazon GameLift Client SDK is bundled with the AWS SDK, which is available in a myriad of different languages. This allows game developers to build game clients with an integration of the Amazon GameLift service in their language of choice.
• Matchmaking: Amazon GameLift continually scans available game servers around the world and matches them against player requests to join games. If low-latency game servers are not available, you can configure the service to automatically add more capacity near your players. Amazon GameLift maintains a queue of waiting players until new games start or new instances launch, then places waiting players into the lowest latency game.
• Player data handling: Game developers can now store custom player information and pass it directly to a game server. A game server or other game entity with an API call can then retrieve Player data from Amazon GameLift.
• Console Support: Amazon GameLift supports games developed and architected for Xbox One and PS4.

Amazon GameLift does the heavy lifting of tasks once required to create session-based multiplayer games by simplifying the process of deploying, scaling, and maintaining game servers while reducing the time, cost, and risks associated with building the infrastructure from scratch.

The reference architecture of a gaming solution that utilizes the Amazon GameLift would look as follows:

Integrating Amazon GameLift into Your Games

The process of integrating Amazon GameLift into your game build can be broken down in a few simple steps:

1. Prepare your game server for hosting on Amazon GameLift by setting up your game server project with the Amazon GameLift Server SDK and adding communication code to the project.
2. Package and upload your game server build to the AWS region targeted for game deployment
3. Create and build a fleet of computing resources to host the game.
4. Prepare your game client to connect to game sessions maintained by Amazon GameLift using the AWS SDK with Amazon GameLift APIs and add code to game client for calls to Amazon GameLift service and identifying the player region.
5. Test your Amazon GameLift integration by connecting an Amazon GameLift-hosted game session and verifying game sessions are being created.

Let’s get started putting these steps into practice by setting up the Amazon GameLift Server SDK in a simple game server project using the Unreal game engine.

Unreal Engine (UE)

We start with Epic’s Unreal game engine. For simplicity, we will create the sample Shooter Game project with online multiplayer functionality built-in, and save it locally on the computer.

Now that I have the Multiplayer Shooter Game sample downloaded and open locally on my machine, I will need to be able to manipulate the C++ code to add the Amazon GameLift service to the UE Online Sub-System to manage the online game sessions. The Shooter Game sample is leveraging the Blueprints Visual Scripting system in Unreal Engine. The Blueprints system is a gameplay scripting system based on node-based interfaces in the UE editor, which enables game designers and content creators to create gameplay elements and functionality within UE editor.

Since it is my goal to use the Amazon GameLift C++ SDK to include the Amazon GameLift service in the game and alter the game code, I will need to create Visual Studio project solution to tie in the game and correlate the source code and any binaries from the Shooter Game to the project. To accomplish this I navigate to the context menu and select the File menu option. In the menu dropdown, I find and select the Generate Visual Studio Project Files option.

Once the project has generated, I only need to return to the Context menu and select File, then Open with Visual Studio in order to open the project and view the source code.

In preparation for adding the Amazon Game Lift service to the Shooter Game as the game service and for game session management, you will need to enable the OnlineSubSystem module in your project. In order to do this, open the game build settings file in the Visual Studio project. Since this game project is named ShooterGame, the build file is named ShooterGame.Build.cs and is located in the Source/ShooterGame folder(s) as shown below.

Open your Build files and uncomment the line for the OnlineSubsystemNull module. Since I am using the sample that already utilizes a multiplayer online system, my build options are set appropriately, and the code looks like this:

public class ShooterGame : ModuleRules
{
	public ShooterGame(TargetInfo Target)
	{
		PrivateIncludePaths.AddRange(
			new string[] { 
				"ShooterGame/Classes/Player",
				"ShooterGame/Private",
				"ShooterGame/Private/UI",
				"ShooterGame/Private/UI/Menu",
				"ShooterGame/Private/UI/Style",
				"ShooterGame/Private/UI/Widgets",
            		}
		);
       PublicDependencyModuleNames.AddRange(
			new string[] {
				"Core",
				"CoreUObject",
				"Engine",
				"OnlineSubsystem",
				"OnlineSubsystemUtils",
				"AssetRegistry",
             			"AIModule",
				"GameplayTasks",
			}
		);
       PrivateDependencyModuleNames.AddRange(
			new string[] {
				"InputCore",
				"Slate",
				"SlateCore",
				"ShooterGameLoadingScreen",
				"Json"
			}
		);
		DynamicallyLoadedModuleNames.AddRange(
			new string[] {
				"OnlineSubsystemNull",
				"NetworkReplayStreaming",
				"NullNetworkReplayStreaming",
				"HttpNetworkReplayStreaming"
			}
		);
		PrivateIncludePathModuleNames.AddRange(
			new string[] {
				"NetworkReplayStreaming"
			}
		);
	}
}

Now that we are set with the Shooter Game project, let’s turn our attention on the Amazon GameLift SDK. I want to leverage the C++ SDK as a plugin for the Unreal Engine, therefore, I need to compile the SDK using the using a compilation directive that builds the binaries for this game engine.

With the SDK source downloaded, I can compile the SDK from the source based upon my operating system. Since I am using a Windows machine for this project, I will complete the following steps:

• Make an out directory to hold the binaries generated from the code compilation:

mkdir out

• Change to the previously created directory:

cd out

• Use CMake to specify a build system generator for VS 2015 Win x64 and set UE compilation flag:

cmake -DBUILD_FOR_UNREAL=1 -G “Visual Studio 14 2015 Win64” <source directory>

• Build C++ project to create binaries using selected Build System (MS Build for this project):

msbuild ALL_BUILD.vcxproj /p:Configuration=Release

With my libraries compiled, I should have the following binary files required to use the Amazon GameLift Unreal Engine plugin.

Linux:

* out/prefix/lib/aws-cpp-sdk-gamelift-server.so

Windows:

* out\prefix\bin\aws-cpp-sdk-gamelift-server.dll

* out\prefix\lib\aws-cpp-sdk-gamelift-server.lib

As you can see below, since I am on Windows, my compiled Amazon GameLift libraries, aws-cpp-sdk-gamelift-server.dll and aws-cpp-sdk-gamelift-server.lib, are located in the prefix\bin and prefix\lib folders respectively.

After copying the binaries to the GameLiftSDK Unreal Engine plugin folder, my Amazon GameLift plugin folder is configured and ready to be added to an Unreal Engine game project.

Given this, it is now time to add the Amazon GameLift plugin to the Unreal Engine ShooterGame project. I could use the Unreal Engine Editor to add the plugin, but instead, I will stay in the Visual Studio project and add the plugin by updating the game directory and project file.

In Windows Explorer, I add a folder called Plugins in the ShooterGame directory and copy my prepared GameLiftServerSDK folder into the directory as noted by the Unreal Engine documentation on plugins.

Now I will open up the ShooterGame.Build.cs file, which is a C# file that holds information about game dependencies.

Within the file I will add the following code:

PublicDependencyModuleNames.AddRange(
            new string[] {
                "Core",
                "CoreUObject",
                "Engine",
                "InputCore",
                "GameLiftServerSDK"
            }
       );

Just to ensure all is in sync with the changes made thus far, I close Visual Studio, go back to the UE Editor, and select Refresh Visual Studio Project.

Upon completion, I select Open Visual Studio and the Plugins folder I added in the ShooterGame directory is now included in the project and able to be viewed in Solution Explorer.

Next, I rebuild my entire solution to get the Amazon GameLift SDK binaries integrated into the project.

I’ll go back to the UE Editor and select Build from the toolbar to ensure the aspects of the Amazon GameLift plugin are included in my ShooterGame. Once compilation is complete, a quick visit to the Settings toolbar and Plugins option shows that the Amazon GameLift plugin is added and is recognized in the project. I will select the Enabled checkbox, which will prompt me to restart the UE Editor. I select Restart Now and allow the Unreal Engine to rebuild the game code files.

Upon completion of the build, the editor will restart and reopen my ShooterGame.

Now things are set for the use of the Amazon GameLift SDK in the ShooterGame project.

With the Unreal editor open, I’ll go into the Open Visual Studio menu option to get back to the ShooterGame code. This will open up Visual Studio and the game code. With Visual Studio open, I go to the ShooterGameMode.cpp file to add the code to initialize the Amazon GameLift SDK. Some key things I must do in order to correctly add the code for Amazon GameLift within my Shooter game project are:

1. Enclose the Amazon GameLift code within a preprocessor condition using the flag WITH_GAMELIFT=1
2. Build a dedicated server in Unreal Engine for my targeted server OS ex. Linux
3. Ensure my build target is a game server type i.e. Type == TargetRules.TargetType.Server

You can find an example of the code needed to add Amazon GameLift in your Unreal Engine project in the documentation here. In addition, you can learn how to build a dedicated server for Unreal Engine by following the Dedicated Server Guide for Windows and Linux provided in the Unreal Engine wiki. With these resources in hand, you should be well on your way to integrating Amazon GameLift into a game project.

I just did a quick review of incorporating the Amazon GameLift SDK in the Unreal Engine game engine, but don’t forget you have the option to add the Amazon GameLift SDK into C# engines like Unity. By downloading the Amazon GameLift Server SDK and compiling the .Net framework 3.5 solution, GameLiftServerSDKNet35.sln. The GameLiftServerSDKNet35.sln solution will enable you to add the Amazon GameLift libraries your Unity3D project. Review the Amazon GameLift SDK documentation, Using the C# Server SDK for Unity, in order to learn more about setting up and using the Amazon GameLift C# Server SDK plugin.

Summary

We reviewed just one of the new aspects added of the Amazon GameLift managed service, but the service provides game developers and game studios with even more. Amazon GameLift enables the building of distributed games by making it easy to manage infrastructure, scale capacity, and match players into available game sessions while defending games from DDoS attacks.

You can learn more about the Amazon GameLift service by reviewing the Amazon GameLift documentation, the Amazon GameLift developer guide and/or check out the Amazon GameLift tutorials on the Amazon GameDev tutorial page in order to hit the ground running with game development with Amazon GameLift service.

Happy Gaming!

– Tara