AWS Official Blog

Using Enhanced RDS Monitoring with Datadog

by Jeff Barr | on | in AWS Lambda, CloudWatch, Guest Post | | Comments

Today’s guest post comes from K Young, Director of Strategic Initiatives at Datadog!

— Jeff;

AWS recently announced enhanced monitoring for Amazon RDS instances running MySQL, MariaDB, and Aurora. Enhanced monitoring includes over 50 new CPU, memory, file system, and disk I/O metrics which can be collected on a per-instance basis as frequently as once per second.

AWS and Datadog
AWS worked closely with Datadog to help customers send this new high-resolution data to Datadog for monitoring. Datadog is an infrastructure monitoring platform that is very popular with AWS customers—you can see historical trends with full granularity and also visualize and alert on live data from any part of your stack.

With a few minutes of work your enhanced RDS metrics will immediately begin populating a pre-built, customizable dashboard in Datadog:

Connect RDS and Datadog
The first step is to send enhanced RDS metrics to CloudWatch Logs. You can enable the metrics during instance creation, or on an existing RDS instance by selecting it in the RDS Console and then choosing Instance Options → Modify:

Set Granularity to 1–60 seconds; every 15 seconds is often a good choice. Once enabled, enhanced metrics will be sent to CloudWatch Logs.

The second step is to send the CloudWatch Log data to Datadog. Begin by setting up a Lambda function to process the logs and send the metrics:

  1. Create a role for your Lambda function. Name it something like lambda-datadog-enhanced-rds-collector and select AWS Lambda as the role type.
  2. From the Encryption Keys tab on the IAM Management Console, create a new encryption key. Enter an Alias for the key like lambda-datadog-key. On the next page, add the appropriate administrators for the key. Next you’ll be prompted to add users to the key. Add at least two: yourself (so that you can encrypt the Datadog API key from the AWS CLI in the next step), and the role created above, e.g. lambda-datadog-enhanced-rds-collector (so that it can decrypt the API key and submit metrics to Datadog). Finish creating the key.
  3. Encrypt the token using the AWS Command Line Interface (CLI), providing the Alias of your just-created key (e.g. lambda-datadog-key) as well as your Datadog keys, available here. Use KMS to encrypt your key, like this: 
    $ aws kms encrypt --key-id alias/ALIAS_KEY_NAME --plaintext '{"api_key":"DATADOG_API_KEY", "app_key":"DATADOG_APP_KEY"}'

    Save the output of this command; you will need it for the next step.

  4. From the Lambda Management Console, create a new Lambda Function. Filter blueprints by datadog, and select the datadog-process-rds-metrics blueprint.
  5. Choose RDSOSMetrics from the Log Group dropdown, enter the Filter Name of your choice, and go to the next page. If you have not yet enabled enhanced monitoring, you must do so before RDSOSMetrics will be presented an as option (see the instructions under Connect RDS and Datadog above):
  6. Give your function a name like send-enhanced-rds-to-datadog. In the Lambda function code area, replace the string after KMS_ENCRYPTED_KEYS with the ciphertext blob part of the CLI command output above.
  7. Under Lambda function handler and role, choose the role you created in step 2, e.g. lambda-datadog-enhanced-rds-collector. Go to the next page, select the Enable Now radio button, and create your function.

That’s It
Once you have enabled RDS in Datadog’s AWS integration tile, Datadog will immediately begin displaying your enhanced RDS metrics. Your RDS instances will be individually identifiable in Datadog via automatically-created tags of the form dbinstanceidentifier:YOUR_DB_INSTANCE_NAME, as well as any tags you added through the RDS console.

You can clone the pre-built dashboard and customize it however you want: add RDS metrics that are not displayed by default, or start correlating RDS metrics with the performance of the rest of your stack.

— K Young, Director of Strategic Initiatives

 

Flatiron Health – Using AWS to Help Improve Cancer Treatment

by Jeff Barr | on | in Guest Post | | Comments

Flatiron Health is a hot startup with a great idea – providing cancer patients, physicians, researchers, and drug firms with a solution that organizes global oncology information. Currently more than 230 cancer clinics and about 2,200 clinicians across the United States use their products and services, which support the treatment of approximately one in five U.S. cancer patients. I asked Alex to tell us about Flatiron’s decision to leverage AWS’s platform. In the guest post below, Alex Lo (Engineering Manager of Developer Infrastructure) tells us how they have put AWS to use!

— Jeff;

Flatiron Health began after our founders witnessed family members and friends battle cancer. They were really frustrated by the inefficiencies in accessing and benefiting from all the diverse and siloed oncology data in fragmented medical record systems. Their vision was to build a disruptive software platform that connects cancer centers across the country, with the goal of supporting deeper insights and understanding to transform how cancer care is delivered.

Time to Market
We’re a startup trying to get our solutions like our OncologyCloud software suite to market as quickly as possible, and our team was unable to iterate and innovate as quickly and as reproducibly as we wanted them to due to a lack of mature automation tools and APIs. We want technology to work for us so we can solve business problems, instead of spending time dealing with computers.

I joined Flatiron in 2015, and fortunately by that time AWS had implemented the healthcare industry compliance standards and processes that we needed. This made us confident that our efforts would be successful. By mid-2015 we began an AWS adoption project. We felt AWS would be the best platform for future growth because of its best-in-class features, rich ecosystem, and excellent HIPAA-eligible features that include encryption, fine-grained security, and auditability. It lets us deliver a unique solution and quickly iterate on our products.

OncoAnalytics, part of Flatiron Health’s OncologyCloud suite, is an analytics tool that unlocks data from multiple systems and delivers detailed clinical insights and business intelligence.

AWS Usage
We use a range of AWS services. Amazon EC2 has advanced features that give us access to virtual machine system logs from the administrative console. Amazon S3 provides us infinitely scaling durable storage, and the encryption features provided with AWS make it straightforward for us to store Protected Health Information (PHI) in S3. We’re running in an Amazon VPC, and use Amazon VPN connections to other networks for secure connectivity. We’re also using AWS IAM, which is wonderful in our environment. It gives us fine-grained security controls so we can enable our engineers to create resources without being full administrators. We use it lot and are experimenting with some of the more advanced features, like the AWS Security Token Service and EC2 Roles. We’re also using auditing and security tools, including AWS Trusted Advisor, AWS CloudTrail, AWS Config, and Amazon CloudWatch.

An additional benefit of AWS is the expertise that we get with the platform. AWS gives really good advice on how to build HIPAA-compliant applications, with account reps specializing in life sciences and Solutions Architects with health tech backgrounds. Plus, AWS has a developer ecosystem that is more mature than what other cloud providers offer. For example, Ansible has an out-of-the-box EC2 inventory module that helps us manage our fleet. We also use both the AWS Command Line Interface and Boto—the AWS SDK for Python—to automate other routine tasks. This automation would be more difficult on other cloud providers.

Almost There
We’re nearing the end of our first consolidation project on AWS. There are a lot of variables and planning involved in moving into a cloud platform and it’s taken us about 10 months, but we expect the migration to be completed by April 2016. Our development teams have been benefitting from the AWS environment for months now, and it’s exciting to see them move faster. We’re looking to leverage AWS in more ways going forward, possibly moving some dedicated hosting assets into the elastic cloud.

To us, the benefits of using AWS are clear. AWS, with its support team, compliance team, tools, ecosystem, and continued feature growth is helping us iterate faster to solve problems that matter in improving cancer care.

Learning More
Read an AWS whitepaper on architecting for HIPAA security and compliance on AWS.

Read a two-part deep-dive article by an AWS solutions architect who worked closely with the Flatiron Engineering team. Part 1 is here and Part 2 is here.

— Alex Lo, Engineering Manager of Developer Infrastructure, Flatiron Health

PS – We are looking for great developers interested in making a difference in the fight against cancer. If you’re interested, get in touch with us.

New – Notifications for AWS CodeCommit

by Jeff Barr | on | in Amazon SNS, AWS CodeCommit, AWS Lambda | | Comments

AWS CodeCommit is a fully-managed source control service that makes it easy for you to host a secure and highly scalable private Git repository.Today we are making CodeCommit even more useful by adding support for repository triggers. You can use these triggers to integrate your existing unit tests and deployment tools into your source code management workflow. Because triggers are efficient and scalable, they are more broadly applicable than a model that is built around polling for changes. I believe that you will find these triggers to be helpful as you move toward a development methodology based on  Continuous Integration and Continuous Delivery.

All About Notifications
You can create up to 10 triggers for each of your CodeCommit repositories. The triggers are activated in response to actions on the repository including code pushes, branch/tag creation, and branch/tag deletion. Triggers can be set to run for a specific branch of a repository or for all branches.

Triggers can send a notification to an Amazon Simple Notification Service (SNS) topic or can invoke a AWS Lambda function. Each trigger can also be augmented with custom data (an uninterpreted string) that you can use to distinguish the trigger from others that run for the same event. You can use triggers to subscribe to repository events through email or SMS. You can wire up SNS to SQS and queue up jobs for your CI/CD tools, or you can use SNS to activate webhooks provided by your tools. In any case, the actions you designate will be triggered by the changes in your CodeCommit repository. You can also use Lambda functions to trigger builds, check syntax, capture code complexity metrics, measure developer producitivity (less is more, of course), and so forth. My colleagues have also come up with some off-unusual ideas that you can find at the end of this post!

You can create, view, and manage your triggers from the AWS Management Console, AWS Command Line Interface (CLI), or via the CodeCommit API.  I used the Console. The navigation column on the left now includes an entry for Triggers:

I simply click on Create Trigger to get started. Then I select the event (or events), pick the branch (or all branches), and fill in the details that are needed to publish a notification or invoke a Lambda function:

Here’s how I choose the events and branches of interest:

 

Then I point to my SNS topic or Lambda function (after ensuring that the proper permissions are in place), use Test Trigger to make sure that it all works as expected, and click on Create.

You can use Test Trigger to verify that your IAM permissions are working as expected. For example, here’s an error that I triggered on purpose:

I fixed this by reading How to Allow AWS CodeCommit to run the Function in the documentation!

Available Now
This new functionality is available now and you can start using it today.To learn more, read about Managing Triggers for an AWS CodeCommit Repository.

My colleague Clare Liguori suggested some creative uses for CodeCommit triggers, above and beyond the usual integration with your CI/CD process:

  • Video Deployment – Have your Lambda function check to see if a new video or a new version of an existing video has been committed, and deploy the video to YouTube.
  • Party Time – Automatically throw and cater a party (using APIs for sandwiches, pizza, and beer) when you deploy a new release.
  • Advertise Releases – When a new release is ready, automatically generate and run a Facebook ad and publicize the release on social media.

I am looking forward to hearing about the creative ways that you make use of these triggers within your development process. Leave me a comment and let me know!

— Jeff;

 

AWS Week in Review – February 29, 2016

by Jeff Barr | on | in Week in Review | | Comments

Let’s take a quick look at what happened in AWS-land last week:

Monday

February 29
Tuesday

March 1
Wednesday

March 2
Thursday

March 3
Friday

March 4
Saturday

March 5
  •  Nothing happened!
Sunday

March 6

New & Notable Open Source

  • LENA is a Lambda Executed NAT Migration Tool.
  • tvarit-maven contains some AWS DevOps automation tools to run Wildfly on OpsWorks.
  • bamboo-ebs is an EBS provisioner for EBS and Bamboo.
  • lambda-packages contains some popular packages, precompiled for use with Lambda.
  • aws-workshop is a set of workshops for AWS, starting with S3.
  • proftpd-mod_aws is an AWS configuration for ProFTPD.
  • keymaker implements lightweight key management on EC2.
  • nubis-nat creates an AWS NAT instance and a Squid proxy.
  • alfresco-cloudformation-chef is a set of CloudFormation templates for the Alfresco One Reference Architecture.
  • humilis-firehose-resource is a custom CloudFormation resource to deploy Kinesis Firehose delivery streams.

New SlideShare Presentations

New Customer Success Stories

  • CrowdStrike – CrowdStrike uses AWS to implement a scalable, cloud-based solution for preventing cyber breaches with on-demand resources, thereby simplifying maintenance, reducing cost, and improving performance.
  • Edmunds.com – After evaluating several Git hosting solutions, Edmunds.com migrated its source code repositories to the cloud on AWS CodeCommit.
  • IXD – By basing its secure email and fax delivery services on AWS, IXD has disrupted the secure document transmission and messaging industry by offering its services for 80 percent less than some of its competitors.
  • Amazon.com – Amazon.com cut costs while improving performance and reliability for its enterprise communications and collaboration software, including Microsoft Exchange, Microsoft Lync, and Microsoft SharePoint.

Upcoming Events

Help Wanted

Stay tuned for next week! In the meantime, follow me on Twitter and subscribe to the RSS feed.

— Jeff;

AWS Import/Export Snowball Update – Export Now Available

by Jeff Barr | on | | Comments

We launched AWS Import/Export Snowball last year at AWS re:Invent (see my blog post, AWS Import/Export Snowball – Transfer 1 Petabyte Per Week Using Amazon-Owned Storage Appliances, to learn more).

At launch time, you had the ability to use this appliance-based model to move large amounts of data (generally 10 terabytes or more) to AWS. This aspect of Snowball is working well and many customers are already making great use of it.

Export Using Snowball
Today, we are making the same model available for data export operations. If you have collected, generated, or stored large terabytes or petabytes of data in AWS and need to get to it more quickly than you could via a network connection, you can now use AWS Import/Export Snowball instead.

You simply log in to the AWS Management Console, create an export request, and specify the data to be exported. A single request can span one or more Amazon Simple Storage Service (S3) buckets. The service will determine how many appliances are needed (each one can hold up to 50 terabytes) and create the export jobs accordingly. The appliances will be prepared, data will be copied to them, and they will be shipped to the address specified in the request. You can track each of these steps using the Console.

As is the case with the existing data import operation, the data stored on the appliance is encrypted using keys that you specify. The keys are not stored on the device.

If your data is stored in Amazon Glacier, you will first need to restore it to S3 (read Restoring Objects to learn more).

When you receive the appliance(s), you simply connect them to your local network, copy the data to local storage, and then tell the service that your are done. The shipping label on the appliance will be updated to display the return address so that you can return the appliance to us.

— Jeff;

 

AWS Week in Review – February 22, 2016

by Jeff Barr | on | in Week in Review | | Comments

Let’s take a quick look at what happened in AWS-land last week:

Monday

February 22
Tuesday

February 23
Wednesday

February 24
Thursday

February 25
Friday

February 26
Saturday

February 27
Sunday

February 28

New & Notable Open Source

New Customer Success Stories

  • The Cleveland Clinic -The Cleveland Clinic runs its Healthy Brains Initiative on AWS and can protect confidential patient data with the built-in security and compliance features of the AWS cloud.
  • National Instruments -National Instruments (NI) reduced testing costs by more than $1 million and shaved months off its software development process by using AWS.
  • Prezi – Using Amazon Redshift and other AWS data services, Prezi has enabled its employees to quickly obtain data to improve the company’s service offerings and help executives identify new revenue opportunities
  • Vinomofo – By launching its website on AWS, Vinomofo was able scale seamlessly to support increases in visitor numbers from 100 during off-peak periods to 12,000 during peak events.

New YouTube Videos

Upcoming Events

Help Wanted

Stay tuned for next week! In the meantime, follow me on Twitter and subscribe to the RSS feed.

— Jeff;

Vodafone DreamLab – Accelerating Cancer Research

by Jeff Barr | on | in Guest Post | | Comments

Continuing what is quickly turning in to series of customer stories, I would like to turn the keyboard over to our friends at Vodafone!

— Jeff;

You may have seen an interesting application called DreamLab circulating on the internet and social media in the past couple of weeks, and we thought we’d give you some insight into how this very innovative application works under the hood.

Fighting Cancer
Cancer touches so many of us; with one in two Australians being diagnosed with cancer by their 85th birthday, medical research is the key to finding better treatments.

Cancer research progress at Garvan Institute of Medical Research is slowed by the limited access researchers have to the supercomputers they need to crunch complex research data.

For a long time, cancer has been thought of by its tissue of origin – for example lung, breast or pancreatic cancer. But cancer is a disease of the DNA, and so with the advances in genome sequencing, world leading researchers at Garvan Institute of Medical Research are interested in creating a library of cancers, grouped based on their genetic mutations.

Analysing Mutations
To do this, researchers need to analyse the genomic mistakes (DNA mutations) of thousands of different cancer patients and group them based on their genetic profiles (not the tissue in which the cancer originated). Garvan has sourced somatic mutations from cancer patients de-identified for research purposes from the published studies of the International Cancer Genome Consortium (ICGC) and The Cancer Genome Atlas (TCGA) to do this. Whilst the research project aims to analyse 24 cancer types / subtypes, the DreamLab app enables users to choose to support breast, pancreatic, ovarian or prostate cancer as the first stage of the project.

The sequencing of just one genome generates tens of gigabytes of data. So huge computing power is needed to complete this analysis. But at Garvan (and around the world) access to supercomputers is limited and costly. That’s where Vodafone’s DreamLab comes in – it enables everyone to donate the computer processing power of their idle Smartphones to speed up cancer research. At the moment the app is available on Android only.

To tackle this problem the Garvan Institute of Medical Research has developed a novel in-house computational algorithm to estimate how functionally close the mutated genes between two patients could be, which can then be used to group tumours based on their mutation profiles, regardless of which tissue the cancer originated.

This algorithm, called the Network Connectivity Analyser (NCA) – was developed by Dr Hong Ching Lee and Dr Jianmin Wu from The Garvan Institute of Medical Research. It has been adapted to run on Android Smartphones for DreamLab.

Inside DreamLab
The algorithm calculates a number of statistics of the interactions between two sets of genes (i.e. mutated genes from two patients).  Within the DreamLab app, the NCA will perform the cross comparison processing whenever the phone is charged to at least 95%. These patterns can help researchers identify subgroups of patients who share similar mutation profiles, and could therefore potentially respond to the same therapies. The combination of a large community (DreamLab users), coupled with big data analytic algorithms, makes this revolutionary form of research possible.

Here’s how DreamLab works:

  1. Garvan uploads their large research problem to Amazon Simple Storage Service (S3).
  2. Once a user has downloaded and set up the DreamLab app, the app will then authenticate against Amazon Cognito and use a set of temporary, limited privilege credentials to request a research job out of Amazon SQS, and then download a small research payload out of S3 (hundreds of KB) with job session state for each phone being managed by Amazon DynamoDB.
  3. A novel algorithm (built by Garvan researchers) in the DreamLab app, allows the phone to solve the research problem, using the phone’s computing power (the algorithm enables the comparison of the functional similarities and differences in mutated genes from different patients, to enable creation of this library of cancers grouped by their genetic profile).
  4. The result is then sent back from the phone to S3, for the Garvan team to analyse.

Informally, this is similar to solving a crossword puzzle with everyone working on a different clue.

Architecture Rationale
Here’s how DreamLab uses AWS:

The nature of the project required an architecture that supports large volumes of data and spiked traffic, whilst remaining cost effective. This required services that can auto scale and/or have no capacity limits. The architecture also needed to support the ability to maintain the state of any data item, whilst simultaneously being updated across more than one client.

Amazon Simple Storage Service (S3) is an optimal service for the storage of data. It has no upper limit in the amount of data that can be stored, and due to its distributed nature it has high redundancy. It also has the added functionality of being able to fire events when an item is added.

Amazon Simple Queue Service (SQS) is a queuing system of unlimited size, supporting the requirement of high traffic and data. The app facing queues utilise the data visibility setting, to ensure that other devices do not fetch an item whilst various operations are taking place.

Amazon DynamoDB is a highly scalable NoSQL database that has no upper limit in record quantity. Whilst the data schema for this system could benefit from a relational database syste,m the benefits of the scale and price of DynamoDB outweigh these.

Amazon Cognito and STS provide substantial security with only a minimal requirement for custom development.

Amazon Elastic Compute Cloud (EC2) servers run all the custom code that is used to maintain the other systems. This custom code is scheduled to run regularly with CRON jobs. The EC2 servers are limited in capacity, but as they are working behind SQS this is not be an issue. We investigated AWS Lambda, a service that runs custom code on an event notification, however Lambda is currently not available in Australia.

The app accesses all SQS queues through Amazon API Gateway. This is so that the communication is over a custom domain that Vodafone uses to zero rate data charges for Vodafone Australia customers in Australia.

By the Numbers
The Garvan Institute currently has 100,000 base datasets files. Each file is 2 MB uncompressed, and 500 KB compressed. In the DreamLab project, each of the base datasets will be downloaded by three users for result validation purposes.

Garvan also has 5,000 analysis tasks, with a size of 1 KB uncompressed and 250 bytes compressed.

Based on the team’s tests, 33 new Android devices will analyse the same amount data about as fast as a CPU core of the Garvan’s supercomputer. The Garvan’s supercomputer consists of 1,280 CPU cores in total.

As of today, DreamLab has over 44,000 active users, which is providing over 1,000 times the processing power when compared to the Garvan’s existing supercomputer allocation for cancer research.

Congratulations & Acknowledgements
Congratulations to Vodafone Foundation, The Garvan Institute of Medical Research and the mobile application partner b2cloud on this great innovation.

You can download DreamLab from the Google Play Store and contribute to this effort if you’d like!

— Andrew Burnet, Domain Delivery Lead, Vodafone Australia

 

 

How Tokyu Hands Architected a Cost-Effective Shopping System with Amazon DynamoDB

by Jeff Barr | on | in Amazon DynamoDB, Customer Success, Guest Post | | Comments

I am a flâneur! I enjoy wandering around a new city, exploring the nooks and crannies, and figuring out what makes it unique and special. On one of my trips to Tokyo I was walking through Shibuya and found and amazing hobby store. The 8-floor building contained tools, supplies, and kits for almost every imaginable hobby. As you can read from the post below (written by my colleagues in Japan), this store, TOKYU HANDS, is now an AWS customer!

— Jeff;

TOKYU HANDS improved its customer experience with an innovative retail point-of-sale and online shopping system. Here’s what Hideki Hasegawa-san (CTO) had to say about their new solution:

As a retailer, we are always conscious about costs and DynamoDB’s easy scalability make it cost-effective to operate. For a few hundred dollars a month, we get a fast, highly-available and scalable database. We don’t need to spend any money on expensive hardware or personnel to operate the database.

Background
TOKYU HANDS is one of Japan’s biggest and most popular retailers. It is a one-stop shop for zakka, Japanese stationary items, creative do-it-yourself (DIY) solutions, and other home items. TOKYU HANDS operates 40 stores across Japan and Singapore. In addition to its retail locations, TOKYU HANDS operates an online store where customers can shop 24/7 using a computer or a mobile device. To keep pace with its rapid growth and new business opportunities, TOKYU HANDS operates a fast, flexible and scalable IT system built entirely on Amazon Web Services (AWS).

Prior to AWS, TOKYU HANDS’ IT systems were located in its on-premises data center. Operating and scaling the data center became a significant burden. So TOKYU HANDS decided to go all-in on AWS and migrated its applications from the on-premises data center to the cloud. Offloading infrastructure management to AWS allowed TOKYU HANDS to focus on delivering more value to its customers. Hasegawa-san, says “I like AWS because we can spend time and resources innovating for our customers, and not on infrastructure management. AWS offers a wide variety of fully managed services that makes it easy for us to architect our entire IT system. Amazon DynamoDB is one such service that is at the core of critical applications.”

The Challenge
TOKYU HANDS’ most important applications for customer experience are its e-commerce system and Point of Sale (POS) + Merchandising System. Its e-commerce system contains all of the business logic to keep the store running. Choosing the right database for the e-commerce system is the key to achieving the scalability, flexibility and customizability needed to match its pace of innovation. TOKYU HANDS’ POS + Merchandising System has a customer-facing application that processes customer orders at the store registers and for online purchases. Among other things, the POS + Merchandising System also keeps track of item inventory as well as customer purchase history. Rather than spending time on routine backend maintenance tasks, TOKYU HANDS wanted its software development team to focus on its customers’ experience by making the POS + Merchandising System better. With its previous architecture, developers spent an inordinate amount of time maintaining and operating the backend data store to support the POS + Merchandising System.

A telling example of the operational burden TOKYU HANDS endured was trying to scale the e-commerce system to handle traffic spikes during “Hands Messe” – an annual super sale similar to Black Friday. The Hands Messe sale generated several times more traffic than usual for TOKYU HANDS’ retail and online stores. In the past, scaling up the database system to handle the spike during Hands Messe was a time-consuming and difficult task. TOKYU HANDS spent a lot time adding, configuring, and operating hardware needed to handle the Hands Messe traffic. Often, they would experience node failures during the sale, resulting in system outages. Even when hardware didn’t fail, TOKYU HANDS experienced sub-optimal database performance. The net result was an inferior shopping experience for their customers and loss of revenue during the 2012 and 2013 seasons. For startups, such service interruptions erodes customer confidence. As a result of their painful experience in managing an on-premises database, TOKYU HANDS began searching for a fully-managed database optimized for availability and scalability.

Here is the Tokyu Hands dev team:

Back row: Hideki Hasegawa, Toshiharu Ozawa, Yoshimitsu Sugawara, Minoru Saito, and Taiji Inoue.

Front row: Seigo Miyoshi, Yusuke Usui, and Manami Osawa.

Burden-free and Cost-effective Operations with DynamoDB
“My first encounter with DynamoDB was at AWS re:Invent. Through presentations and conversations with solution architects, we learned about DynamoDB’s capabilities and were intrigued by its high availability, scalability and worry-free operations. When I learned that core applications for Amazon.com and other AWS services were using DynamoDB, I decided to give it a try”, said Hasegawa-san, explaining how they started using Amazon DynamoDB.

In just a few months, TOKYU HANDS was able to re-architect its e-commerce system to use Amazon DynamoDB. In 2014, TOKYU HANDS put the new e-commerce system to the test with its annual Hands Messe sale. Amazon DynamoDB’s multiple availability zone architecture ensured their tables were highly available. TOKYU HANDS did not have to worry about system down-time resulting from node failures. TOKYU HANDS was also able to handle traffic spikes easily by scaling up DynamoDB throughput just before the sale.  Unlike previous years, none of the database requests were rejected due to capacity constraints. After the sale ended, TOKYU HANDS dialed back throughput, thereby saving them money. TOKYU HANDS is considering purchasing reserved capacity for throughput in order to save even more.

Easy Application Development with Amazon DynamoDB
Using Amazon DynamoDB together with a host of other AWS services including Amazon S3, Amazon SQS, and Amazon SNS, TOKYU HANDS was able architect a brand new POS + Merchandising System. “Most of our team is made up of former clerks at our retail stores. They know our customers in and out, and we want them to use that knowledge to build powerful applications and not worry about infrastructure. With AWS, we are able to do just that”.

Here is a diagram of their architecture:

Summary
TOKYU HANDS has experienced three important benefits with AWS and Amazon DynamoDB:

  • Cost effective operations
  • Hands-off, worry-free operations
  • Easy development of high-availability applications

See how your business can leverage this fully managed AWS NoSQL service to achieve cost-effective scale by visiting our DynamoDB Getting Started and Developer Resources pages.  Then head over to the FAQs to learn how to leverage DynamoDB Streams and Triggers using serverless programming with AWS Lambda.

— AWS Japan Solution Architects

 

How The Healthcare of Tomorrow is Being Delivered Today

by Jeff Barr | on | in Guest Post | | Comments

My colleagues Jessica Beegle and Angel Pizarro wrote the guest post below to introduce you to an exciting new approach to healthcare!

— Jeff;

Precision Medicine is an emerging approach for disease prevention and treatment that takes into account a person’s individual variation in genes, lifestyle, and environment.

At AWS, we are proud to support the White House Precision Medicine Initiative and will continue our work to accelerate innovation and reduce time to translate research into clinical practice by providing a highly secure platform which can connect researchers and clinicians across the globe, making high-value healthcare datasets such as the 1,000 Genomes, The Cancer Genome Atlas (“TCGA”), and the International Cancer Genome Consortium (“ICGC”) available for use by qualified researchers free-of-charge, and supporting innovative global challenges such as the Qualcomm Tricorder XPRIZE and the Digital Mammography DREAM Challenge.

AWS has worked with many leading global enterprises, innovative startups, and top regulators over the past few years and below are a few highlights from our ecosystem of partners who are bringing precision medicine out of the lab and into clinical care, all with the aim of delivering the healthcare of tomorrow, today.

NCI-MATCH Clinical Trial and Thermo Fisher Scientific
The National Cancer Institute Molecular Analysis for Therapy Choice (NCI-MATCH) program is defining a new paradigm for how cancer could be addressed in the era of personalized medicine. The NCI-MATCH program is the first nation-wide trial in which targeted next-generation sequencing (NGS) technology is being used to screen tumor samples for multiple genetic mutations simultaneously in an effort to place patients in one of up to 20 treatment arms that are part of this study.

Multi-arm trial designs like NCI-MATCH allow researchers to cast a wider net, which helps take into account relatively rare tumor mutations and can drive the development of promising new therapies. The leaders of the program selected AWS partner Thermo Fisher Scientific’s Ion Torrent NGS system and complementary Oncomine reagents for its ability to deliver robust data from limited amounts of starting material derived from formalin fixed paraffin-embedded samples.

Mark Stevenson (Executive Vice President and President, Life Sciences Solutions, for Thermo Fisher) told us:

A study of this scale would not be feasible using the traditional one-sample, one-biomarker testing approach. It is the first national oncology trial of its kind being conducted in the spirit of President Obama’s Precision Medicine Initiative, which has the potential to transform the future of cancer care.

Led by the NCI, the trial is a first-of-its-kind collaboration with multiple pharmaceutical companies.  More than 3,000 samples are being screened at four sites across the country using a standardized sequencing protocol on the NGS system to ultimately enroll up to 1,000 patients in the trail.

The sequencing sites participating in the program include:

  • NCI Molecular Characterization Laboratory (Frederick, Maryland)
  • The University of Texas MD Anderson Cancer Center (Houston, Texas)
  • Massachusetts General Hospital
  • Yale University (New Haven, Connecticut)

Using the sequencing results, the study’s leads assign program participants, if eligible, to one of several trial arms based on the genetic alterations associated with their tumor, rather than their cancer type.

Precision Medicine Software Makes Genomic Data Actionable: Syapse
The use of genomic and molecular data has become common practice in the diagnosis of disease subtypes and in determining appropriate therapeutic treatment paths, yet clinicians are often unprepared to use these data for clinical decision-making. Traditional Electronic Health Records (EHRs) and other IT systems lack the capability to handle the scale and complexity of genomic and molecular data, and make the data useful for physicians.

To solve this problem facing many health systems and clinicians, Syapse created a software platform that integrates patients’ clinical, treatment, and outcomes data with genomic and molecular data. Using the Syapse Precision Medicine Platform, physicians can derive actionable insights from genomics and molecular data at the point-of-care. Syapse has been adopted by many leading healthcare organizations including Intermountain Healthcare, Providence Health and Services, Sarah Cannon, UCSF, Stanford University, and Sanford Health.

A recent study performed by Intermountain Healthcare produced evidence that their precision medicine program, supported by Syapse software. Here’s what it said:

Precision cancer medicine appears to significantly improve survival for patients with advanced cancer when compared to control patients who received conventional chemotherapy. The additional survival is not associated with increased costs. While the results of this study warrant further investigation in the setting of a prospective randomized control trial, this genomics-based approach appears to be a viable, and perhaps superior, option for patients with advanced or metastatic cancer.

You can watch this video to learn more about how Intermountain and Syapse worked together:

iHART: Decoding the Causes of Autism
AWS and Illumina partnered with researchers from the Hartwell Autism Research and Technology Initiative (iHART) to establish the largest open-access repository of bioinformatics data on autism spectrum disorders through Illumina’s cloud-based platform, BaseSpace. The collaborative research effort includes researchers from Stanford University, the University of California Los Angeles, and the New York Genome Center, who will be granted access to 5,000 genomes of both individuals with autism and those of their family members from the National Institute of Mental Health. Researchers will be able to access BaseSpace and Amazon Redshift to quickly analyze this valuable data to determine genomic variations across the samples.

Dr. Dennis Wall is Associate Professor of Pediatrics at the Stanford University School of Medicine and iHART’s principal investigator. He told us:

We believe this landmark collaboration and open initiative will set the stage for major, clinically useful discoveries in the near future. The complexity of autism requires big data scientific initiatives like this that are openly accessible and act as a sandbox in which all qualified researchers can play. Ultimately, we hope our effort will help define the forms of autism and bring sufficient clarity for marker development and much more.

PrecisionFDA: Advancing Innovation Through Collaboration
President Obama envisioned individualizing a diagnosis, treatment or even a cure for a person based on their genes when he announced his Precision Medicine Initiative a year ago.  FDA’s precisionFDA is a step closer to achieving that vision. PrecisionFDA offers an online, cloud-based portal which creates a sandbox for scientists, academia, government and others across the genomics community to experiment, share data and tools, collaborate on projects, and define new standards for evaluating analytical software.

PrecisionFDA was created by the FDA’s Office of Health Informatics (OHI) under FDAs’ Chief Health Informatics Officer, Taha Kass-Hout, MD, MS, and OHI’s Deputy Director, Elaine Johanson, and is being developed under a contract with DNAnexus. The goal of precisionFDA is to help advance the regulatory science needed to assess the accuracy of genome testing and software. The creation of a secure cloud-based platform which is both open and transparent allows members of the genomics community at large to test, develop, and explore next-generation sequencing (“NGS”) methodologies to spur innovation needed to develop necessary standards. In addition, the creation of new reference datasets will unite the genomics community around common standard resources for the industry, and in doing so, will advance consumer safety.

Since it debuted on Dec 15, 2015, more than 900 members representing over 430 organizations worldwide have joined precisionFDA.  Significantly, more than 130 members have shared software or reference genome material with the rest of the precisionFDA community. While U.S. members account for approximately 2/3 of the membership, almost half come from Silicon Valley. In addition, precisionFDA’s source code was publicly released on GitHub in January 2016.

Richard Daly (CEO of DNAnexus) told us:

DNAnexus is proud to be delivering precisionFDA and creating a community around open-source genomic analysis pipelines, reference data, and analytical processing resources. The FDA has taken a leadership position in making President Obama’s Precision Medicine Initiative a reality, and the DNAnexus platform will enable the managing and sharing of genomic data at an unprecedented level.

Here’s a video with more information about DNAnexus and their participation in precisionFDA:

NCI Cancer Genomics Cloud Pilot & Seven Bridges Genomics
This pilot aims to increase biomedical discovery by democratizing access to the world’s largest cancer genomics datasets.

Seven Bridges Genomics was selected as a winner of the National Cancer Institute’s Cancer Genomics Cloud Pilot, an initiative to develop a cloud-based infrastructure for data sets such as the Cancer Genome Atlas, which will exceed 2.5 PB, with the goal of increasing the speed of discovery through the democratization of access to some of the world’s most valuable cancer genomic datasets. While the quantity of genomic data has increased dramatically over the last few years, researchers lack the proper resources and methods to access, disseminate, share, and mine data at this scale today. The NCI Cancer Genomics Cloud is being created to solve these challenges and support co-localized data and computation in order to more effectively integrate the big data of cancer genomics.

Deniz Kural (Founder and CEO of Seven Bridges Genomics) told us:

With more than 14 million people in the US living with cancer, it is critical that we build a solution that matches the scale of the challenge. The NCI Cancer Genomics Cloud will enable cancer researcher to utilize genomic data and work towards making personalized medicine a reality. Our platform democratized genomics research and allows anyone with an Internet connection to access and analyze these massive datasets.

Learn More
We hope that you have enjoyed this review of some of the ways that our partners are using the AWS Cloud to solve some of today’s toughest and most critical healthcare challenges. To learn more, please visit our Genomics in the Cloud page and our Healthcare Providers and Insurers in the Cloud page.

— Jessica Beegle , Global Leader, Healthcare and Life Science Partner Ecosystem, AWS
— Angel Pizarro, Global Technical Business Development, Scientific Computing, AWS

 

AWS Marketplace Adds SharePoint Enterprise

by Jeff Barr | on | in AWS Marketplace | | Comments

AWS Marketplace makes it simple for you to discover, buy, and deploy many different types of software packages and applications including Network Infrastructure, Security, Web Apps, Business Intelligence, Big Data, DevOps, Media, and Storage / Backup.

Access to SharePoint Enterprise
I am happy to be able to announce that SharePoint Enterprise is now available to purchase and deploy in AWS Marketplace. Designed to suit the needs of many different types and sizes of organizations up to and including multinational enterprises, SharePoint Enterprise supports custom applications, mobile applications, social media integration, business intelligence, and advanced content management (none of which is available in the entry-level SharePoint Foundation product).

This new offering helps customers to migrate their SharePoint and Windows Server workload and is made possible by the folks at Data Resolution. As a long-time SharePoint provider and Microsoft Hosting Provider, they have the background and experience needed to design and support a scalable implementation of SharePoint Enterprise in the AWS Cloud. They are already migrating one of their on-premises SharePoint customers to AWS, and are planning to work with us to produce a customer success story later this year. Data Resolution will also be offering consulting and migration services in order to make your transition to the cloud as smooth as possible.

As you will see when you launch SharePoint on your own, this implementation makes use of our new AWS Marketplace Support for Clusters and AWS Resources. The sophistication and size of the AWS implementation varies according to the product type. For example, the SharePoint Enterprise 2013 Advanced option includes single-server deployments of Active Directory 2012 and SQL Server 2012, along with a load-balanced pair of instances running Windows Server 2012.

Multiple Options
We have worked with Data Resolution to make sure that you have several different options when you choose to run SharePoint Enterprise on AWS (all of the options include phone, email, and forum support from Data Resolution):

  • Product Type – You can choose SharePoint Enterprise 2013 Basic, SharePoint Enterprise 2013 Business, or SharePoint Enterprise 2013 Advanced.
  • Pricing Model – You can start with a free trial,  pay by the hour, by the month, or annually. If you choose the annual option, the pricing reflects a 10% discount.
  • User Count – You can choose between 10, 25, and 100 user options (in this context, a user refers to someone who adds, deletes, or changes content stored in SharePoint).
  • License Type – If you already have licenses for SharePoint Enterprise 2013 Business or SharePoint Enterprise 2013 Advanced, you can make use of them as part of this offering. You can also choose an option that includes the cost of the applicable license.
  • AWS Region – You can run SharePoint Enterprise in the US East (Northern Virginia), US West (Oregon), or US West (Northern California) Regions.

These options give you a lot of flexibility! You can run a small deployment of SharePoint Enterprise 2013 Basic on an hourly basis for development and testing, or to meet the needs of a short-term project or event. Or, you can run a larger deployment of SharePoint Enterprise 2013 Advanced on a permanent basis. Either way, you can choose to run in one of three US locations. All of these options will allow you to address and accommodate success and growth in usage without undue hassles.

Getting Started
You can get started at no charge by launching the free trial of SharePoint Enterprise 2013 for AWS Basic “All in One.” This offering was designed for test and development teams working on short-term projects.

— Jeff;