AWS Official Blog

My colleague Jon Fritz wrote the guest post below to introduce you to the newest version of Amazon EMR.

— Jeff;

Today we are announcing Amazon EMR release 4.2.0, which adds support for Apache Spark 1.5.2, Ganglia 3.6 for Apache Hadoop and Spark monitoring, and new sandbox releases for Presto (0.125), Apache Zeppelin (0.5.5), and Apache Oozie (4.2.0).

New Applications in Release 4.2.0
Amazon EMR provides an easy way to install and configure distributed big data applications in the Hadoop and Spark ecosystems on managed clusters of Amazon EC2 instances. You can create Amazon EMR clusters from the Amazon EMR Create Cluster Page in the AWS Management Console, AWS Command Line Interface (CLI), or using a SDK with EMR API. In the latest release, we added support for several new versions of applications:

• Spark 1.5.2 – Spark 1.5.2 was released on November 9^th, and we’re happy to give you access to it within two weeks of general availability. This version is a maintenance release, with improvements to Spark SQL, SparkR, the DataFrame API, and miscellaneous enhancements and bug fixes. Also, Spark documentation now includes information on enabling wire encryption for the block transfer service. For a complete set of changes, view the JIRA. To learn more about Spark on Amazon EMR, click here.
• Ganglia 3.6 – Ganglia is a scalable, distributed monitoring system which can be installed on your Amazon EMR cluster to display Amazon EC2 instance level metrics which are also aggregated at the cluster level. We also configure Ganglia to ingest and display Hadoop and Spark metrics along with general resource utilization information from instances in your cluster, and metrics are displayed in a variety of time spans. You can view these metrics using the Ganglia web-UI on the master node of your Amazon EMR cluster. To learn more about Ganglia on Amazon EMR, click here.
  
• Presto 0.125 – Presto is an open-source, distributed SQL query engine designed for low-latency queries on large datasets in Amazon S3 and the Hadoop Distributed Filesystem (HDFS). Presto 0.125 is a maintenance release, with optimizations to SQL operations, performance enhancements, and general bug fixes. To learn more about Presto on Amazon EMR, click here.
• Zeppelin 0.5.5 – Zeppelin is an open-source interactive and collaborative notebook for data exploration using Spark. You can use Scala, Python, SQL, or HiveQL to manipulate data and visualize results. Zeppelin 0.5.5 is a maintenance release, and contains miscellaneous improvements and bug fixes. To learn more about Zeppelin on Amazon EMR, click here.
  
• Oozie 4.2.0 – Oozie is a workflow designer and scheduler for Hadoop and Spark. This version now includes Spark and HiveServer2 actions, making it easier to incorporate Spark and Hive jobs in Oozie workflows. Also, you can create and manage your Oozie workflows using the Oozie Editor and Dashboard in Hue, an application which offers a web-UI for Hive, Pig, and Oozie. Please note that in Hue 3.7.1, you must still use Shell actions to run Spark jobs. To learn more about Oozie in Amazon EMR, click here.
  

Launch an Amazon EMR Cluster with Release 4.2.0 Today
To create an Amazon EMR cluster with 4.2.0, select release 4.2.0 on the Create Cluster page in the AWS Management Console, or use the release label emr-4.2.0 when creating your cluster from the AWS CLI or using a SDK with the EMR API.

— Jon Fritz, Senior Product Manager

Earlier this year, my colleague Peter Moon shared our plans to launch an AWS SDK for Go. As you will read in Peter’s guest post below, the SDK is now generally available!

— Jeff;

At AWS, we work hard to promote and serve the developer community around our products. This is one of the reasons we open-source many of our libraries and tools on GitHub, where we cherish the ability to directly communicate and collaborate with our developer customers. Of all the experiences we’ve had in the open source community, the story of how the AWS SDK for Go came about is one we particularly love to share.

Since the day we took ownership of the project 10 months ago, community feedback and contributions have made it possible for us progress through the experimental and preview stages, and today we are excited to announce that the AWS SDK for Go is now at version 1.0 and recommended for production use. Like many of our projects, the SDK follows Semantic Versioning, which means starting from 1.0, you can upgrade the SDK within the same major version 1.x and have confidence your existing code will continue to work.

Since the Developer Preview announcement in June, we have added a number of key improvements to the SDK, including:

• Sessions – Easily share configuration and request handlers between clients.
• JMESPATH support – Query and reshape complex API responses and other structures using simple expressions.
• Paginators – Iterate over multiple pages of list-type API responses.
• Waiters – Wait for asynchronous state changes in AWS resources.
• Documentation – Revamped developer guide.

Here’s a code sample that exercises some of these new features:

// Create a session
s := session.New(aws.NewConfig().WithRegion("us-west-2"))
// Add a handler to print every API request for the session
s.Handlers.Send.PushFront(func(r *request.Request) {
	fmt.Printf("Request: %s/%s\n", r.ClientInfo.ServiceName, r.Operation)
})
// We want to start all instances in a VPC, so let's get their IDs first.
ec2client := ec2.New(s)
var instanceIDsToStart []*string
describeInstancesInput := &ec2.DescribeInstancesInput{
	Filters: []*ec2.Filter{
		&ec2.Filter{
			Name:   aws.String("vpc-id"),
			Values: aws.StringSlice([]string{"vpc-82977de9"}),
		},
	},
}
// Use a paginator to easily iterate over multiple pages of response
ec2client.DescribeInstancesPages(describeInstancesInput,
	func(page *ec2.DescribeInstancesOutput, lastPage bool) bool {
		// Use JMESPath expressions to query complex structures
		ids, _ := awsutil.ValuesAtPath(page, "Reservations[].Instances[].InstanceId")
		for _, id := range ids {
			instanceIDsToStart = append(instanceIDsToStart, id.(*string))
		}
		return !lastPage
	})
// The SDK provides several utility functions for literal <--> pointer transformation
fmt.Println("Starting:", aws.StringValueSlice(instanceIDsToStart))
// Skipped for brevity here, but *always* handle errors in the real world :)
ec2client.StartInstances(&ec2.StartInstancesInput{
	InstanceIds: instanceIDsToStart,
})
// Finally, use a waiter function to wait until the instances are running
ec2client.WaitUntilInstanceRunning(describeInstancesInput)
fmt.Println("Instances are now running.") 

We would like to again thank Coda Hale and our friends at Stripe for contributing the original code base and giving us a wonderful starting point for the AWS SDK for Go. Now that it is fully production-ready, we can’t wait to see all the innovative applications our customers will build with the SDK!

For more information please see:

• The AWS SDK for Go repository.
• API Documentation.
• Developer Guide.

Peter Moon, Senior Product Manager

If you build mobile apps, you know that you have two implementation choices. You can build native or hybrid applications that compile to an executable file. You can also build applications that run within the device’s web browser.

We launched the AWS Device Farm in July with support for testing native and hybrid applications on iOS and Android devices (see my post, AWS Device Farm – Test Mobile Apps on Real Devices, to learn more).

Today we are adding support for testing browser-based applications on iOS and Android devices. Many customers have asked for this option and we are happy to be able to announce it. You can now create a single test run that spans any desired combination of supported devices and makes use of the Appium Java JUnit or Appium Java TestNG frameworks (we’ll add additional frameworks over time; please let us know what you need).

Testing a Web App
I tested a simple web app. It opens amazon.com and searches for the string “Kindle”. I opened the Device Farm Console and created a new project (Test Amazon Site). Then I created a new run (this was my second test, so I called it Web App Test #2):

Then I configured the test by choosing the test type (TestNG) and uploading the tests (prepared for me by one of my colleagues):

The file (chrome-with-screenshot.zip) contains the compiled test and the dependencies (a bunch of JAR files):

Next, I choose the devices. I had already created a “pool” of Android devices, so I used it:

I started the run and then checked in on it a few minutes later:

Then I inspected the output, including screen shots, from a single test:

Available Now
This new functionality is available now and you can start using it today! Read the Device Farm Documentation to learn more.

— Jeff;

My colleagues Angel Pizarro and Ariel Gold wrote the incredibly interesting guest post below.

— Jeff;

Today we are pleased to announce that qualified researchers can now access two of the world’s largest collections of cancer genome data at no cost on AWS as part of the AWS Public Data Sets program. Providing access to these petabyte-scale genomic data as shared resources on AWS lowers the barrier to entry, thus expanding the research community and accelerating the pace of research and discovery in the development of new treatments for cancer patients.

The Cancer Genome Atlas (TCGA) corpus of raw and processed genomic, transcriptomic, and epigenomic data from thousands of cancer patients is now freely available on Amazon S3 to users of the Cancer Genomics Cloud, a cloud pilot program funded by the National Cancer Institute and powered by the Seven Bridges Genomics platform.

The International Cancer Genome Consortium (ICGC) PanCancer dataset generated by the Pancancer Analysis of Whole Genomes (PCAWG) study is also now available on AWS, giving cancer researchers access to over 2,400 consistently analyzed genomes corresponding to over 1,100 unique ICGC donors. These data will also be freely available on Amazon S3 for credentialed researchers subject to ICGC data sharing policies.

These two data sets represent the first controlled-access genomic data that have been redistributed to the wider research audience on the cloud. Previously, researchers needed to download and store their own copies of the data before they could begin their experiments. Now, with this data hosted on AWS for the community, researchers can begin their work right away. Researchers will also have access to a broader toolset hosted and shared by the community within AWS. This translates into a much lower barrier to entry and more time for science.

Making these data and tools available in the cloud will also enable a greater level of collaboration across research groups, since they will have a common place to access and share data. Finally, researchers will also be able to securely bring their own data and tools into AWS, and combine these with the existing public data for more robust analysis. No-cost data access, a broader set of available tools, and increased collaborative capabilities will enable researchers to focus on their science and not infrastructure, allowing them to get more done in shorter periods, and ultimately accelerating the pace of research and discovery in the study of cancer.

Accessing TCGA and ICGC on AWS
The difference between TCGA and ICGC, and previously released AWS Public Data Sets such as the National Institutes of Health (NIH) 1000 Genomes Project, Genome in a Bottle (GIAB), and the 3000 Rice Genome, is the need to limit access to researchers that have gone through a review process for their intended use of the data. Because of this requirement, access to TCGA and ICGC on AWS will be administered by our third-party partners, Seven Bridges Genomics and the Ontario Institute for Cancer Research, respectively. These partners have the rights to redistribute the data on behalf of the original data providers. The partners will also curate and update the data over time, as well as develop a community of users who can share cloud-based tools and best practices in order to accelerate use of the data and advance our understanding of cancer.

You can learn more about the data sets, and specifics on how to access them, on our TCGA on AWS page and ICGC on AWS page.

Tools and Resources for Working with the Data
The TCGA data will be available to users of the Cancer Genomics Cloud (CGC). Researchers can apply for early access here. Once accepted, users will be able access the data via the CGC Web portal or use the CGC’s API for programmatic access to the data. The CGC will have a set of data analysis pipelines already integrated into the platform so that users can start working right away with the most common toolsets.

The ICGC data will be generally accessible via the use of a downloadable command line tool. Users can search for files using the ICGC Data Portal and access individual or related sets of alignment and variant files through the ICGC Storage Client. The alignments and a selection of Sanger somatic variant calls are currently available in Amazon S3. Further variant calls will be released following additional quality checking, validation, and analysis. For more information see the ICGC on the Cloud page and ICGC Storage Client documentation.

As always, when working with sensitive genomics data on AWS, you should take care to secure your storage and computational resources. The Architecting for Genomic Data Security and Compliance in AWS whitepaper is a good starting point if you are unfamiliar with the service features and tools necessary to work with data in a secure manner. Genomics platforms such as the CGC take care to meet these types of requirements as part of their value proposition. For example, DNAnexus has provided user documentation on how to leverage the ICGC Storage Client within their platform here.

Recognizing that it is no easy task to work with data at this scale, the PCAWG group are also releasing the PanCancer Launcher. This is an open source system to create EC2 instances, enqueue the analysis work items, trigger Docker-based analysis pipelines, and clean up the launched resources as computational tasks complete.

Currently, the PanCancer Launcher includes support for the BWA-mem-based alignment pipeline and its associated quality control steps.  Future releases will expand support for the variant calling pipelines created by the project that encompassed current best practice variant calling pipelines from 4 academic organizations: the German Cancer Research Center (DKFZ), the European Molecular Biology Laboratory (EMBL) in Heidelberg, the Wellcome Trust Sanger Institute, and the Broad Institute. You can read more about how to leverage the PanCancer Launcher in the Launcher HOWTO Guide.

Genomics in the Era of Cloud Computing
It has been interesting to witness the parallel evolution of genomics and cloud computing over the last decade. Both have been driven by new technologies that leverage economies of scale. Both have fundamentally changed the types of questions that can be asked simply because we can now collect and analyze the data in the same place.

The genomics research community, which have witnessed their storage and compute requirements double overnight when new chemistry kits are released, realized long ago that scalable cloud computing models are a better fit than large capital purchases that have to be planned for and amortized over 3-5 years. Today, it is common practice to work with data sets that reach in the hundreds of terabytes, and a few important ones that reach into the petabytes like the TCGA and ICGC. For genomics, cloud has become the new normal for how science gets done.

You can learn more about how genomics thought leaders are innovating in the genomics field through the use of cloud in this new video:

Be sure to also visit the Scientific Computing on AWS and Genomics on AWS pages for more user stories and tools.

Thank You
We’d like to thank our collaborators at the Ontario Institute for Cancer Research and Seven Bridges Genomics who helped us launch these public data sets and will be curating the data, administering access, and cultivating the ecosystem of tools around them. We look forward to working with many more organizations and researchers who will share their expertise and tools in order to accelerate the development of new treatments for cancer patients. Tell us how you’re using the data via the TCGA on AWS and ICGC on AWS pages and sign up for project updates.

— Angel Pizarro (AWS Scientific Computing) and Ariel Gold (AWS Public Data Sets)

Let’s take a quick look at what happened in AWS-land last week:

New & Notable Open Source

• AWSTOOLS is a set of tools for AWS.
• image_upload_AmazonS3_Rails uploads images to S3 using Rails, Paperclip, and Imagemagick.
• aws-monocyte destroys your AWS resources in non-European regions (use with care).
• Prescience sets up New Relic monitoring for Elastic Beanstalk.
• aws-lambda-local runs Lambda functions locally.
• kinesis-connector is a Kinesis Connector Plugin for the Reekoh IoT platform.
• aws_role_credentials generates AWS credentials for roles using STS.
• node-aws-lambda helps to automate deployment of AWS Lambda functions.
• ci_lambda_checks is the Cloud Insight Lambda Framework.
• Cloudwatch-Loggly retrieves CloudWatch logs and sends them to Loggly.
• simplerobotservice uses the AWS IoT service to connect a Leap Motion controller to a robot arm.
• aws-bootcamp provides materials for AWS Bootcamps.
• alpha is a continuous deployment tool for Lambda.
• aws-iot-toolkit contains a set of tools and files to configure AWS IoT.
• sheepherding is AWS reporting driven by Lambda scripts.
• reservedinstances is a Reserved Instance analysis and recommendations tool.
• awspec is a set of RSpec tests for your AWS resources.
• lambda-gateway-twilio-demo uses Twilio to save an image from your mobile phone to AWS.
• Minio Go is a library for S3-compatible cloud storage.
• cloudogram is a Meteor app for monitoring AWS infrastructure.
• aws-tools is a set of tools for quick & easy AWS administration.
• aws-ps-scripts contains a set of AWS PowerShell scripts.
• datapackage-validator-awslambda is a Lambda function for validating bulk-loaded data packages (works with bawlk).
• domchk53 is a domain availability checker that uses the Route 53 API.
• igor manages the provisioning of AWS resources and also helps with deployment.

New SlideShare Presentations

• Earth Observation in the Cloud.
• Digital Globe Presentation for Earth Observation in the Cloud Demo Day.
• developmentSEED Presentation for Earth Observation in the Cloud Demo Day.
• Real-Time Event Processing.
• Big Data Collection & Storage.
• Cloud HPC at AWS.
• Data Processing and Analytics.
• AWS Big Data Platform.
• Increase Speed and Agility with Amazon Web Services.
• Amazon Redshift Masterclass.
• Protecting Your Data in AWS.
• Network Security & Access Control in AWS.
• Account Separation and Mandatory Access Control on AWS.
• Shared Responsibility Deep Dive.
• Cloud Security Guidance from CESG and AWS.
• Event-Driven Programming with DynamoDB Streams and AWS Lambda.
• IoT Hack Day: AWS Pop-up Loft Hack Series.
• Licensing considerations for Enterprise Applications in the AWS Cloud.
• Best practices: Backup and Recovery for Windows Workloads.
• Running Active Directory in the AWS Cloud.
• AWS LA Media & Entertainment Event:
  • Shoot the Bird: Linear Broadcast on AWS.
  • Cloud Analytics for Audience Engagement.
  • Security of Digital Media Content & Applications on AWS.
• AWS Enterprise Summit Manila:
  • Welcome.
  • Enterprise Innovations.
  • Understanding Security and Compliance on AWS.
• AWS Seminar Series 2015:
  • Auckland.
  • Brisbane.
  • Sydney.
• AWS October Webinar Series:
  • Introducing Amazon QuickSight.
  • Introducing AWS Import / Export Snowball.
  • Introduction to AWS WAF.
  • Getting Started with AWS IoT.
  • Introducing Amazon Kinesis Firehose.
  • Introducing AWS Mobile Hub.
  • Using Spot Instances to Save up to 90% off Your EC2 Bill.
  • AWS Lambda Best Practices: Python, Scheduled Jobs, and More.

New Customer Success Stories

• 3DVIA.
• Kuoni France.
• OpenClassrooms.
• Veolia Water France.
• Aire.
• Beatpacking.
• The BMW Group.
• BuildFax.
• Burt.
• Canary.
• Capital One.
• The Center for American Progress.
• Decisyon.
• Eataly.
• General Electric.
• Go Energy.
• Holiday Extras.
• Interflora.
• itsNOON.
• John Deere.
• Krux.
• LiteracyPlanet.
• MLB Advanced Media.
• Manipal Global.
• Matchmove.
• Omise.
• OutSystems.
• Pinterest.
• PocketMath.
• Remind.
• SEAT Pagine.
• Securitas Direct.
• Segment.
• Stripe.
• The Seattle Times.
• Tictail.
• Trax.
• Unali.
• The University of Pennsylvania.
• Talen Energy.

New YouTube Videos

• AWS Elastic Beanstalk Deployment Walkthrough.
• What is Amazon DynamoDB?

Upcoming Events

• AWS Webinars for November 2015 – New Services and Best Practices.
• November 20 (Partner Webinar)  –The Reserved Instance lifecycle: Maximizing Savings Post-Purchase.
• December 1 (Meetup in Chicago, IL)  – APIs and IPAs.
• AWS Lofts:
  • San Francisco.
  • New York.

Help Wanted

• Stelligent – DevOps Automation Engineer (Advanced, Senior, Principal).
• Senior Leader: AWS VPC (Virtual Private Cloud).
• EC2 Systems Engineering Leader.
• Senior Software Development Manager, EC2 Networking.
• AWS Careers.

Stay tuned for next week! In the meantime, follow me on Twitter and subscribe to the RSS feed.

— Jeff;

I have been traveling and got behind on my Week in Review posts. Here’s the day-by-day for late October’ I’ll catch up on everything else (code, presentations, videos, events, and so forth) in the post for the week of November 9th.

— Jeff;

I have been traveling and got behind on my Week in Review posts. Here’s the day-by-day for late October; I’ll catch up on everything else (code, presentations, videos, events, and so forth) in the post for the week of November 9th.

— Jeff;

Today we have a very special guest post. My colleague Shannon Kellogg is a senior member of our Public Policy team. He has an interesting perspective on the role of public policies for renewable energy, with a focus on how they affect investment and economic growth.

— Jeff;

In late September, I traveled to Virginia’s Eastern Shore to join Governor Terry McAuliffe and other state and local officials for his announcement of the Commonwealth’s issuance of its first permit for a new 80 megawatt solar facility that will be constructed in Accomack County. Amazon Web Services (AWS) has partnered with Community Energy to build the new installation — Amazon Solar Farm US East — and the project is expected to more than quadruple the amount of solar energy currently installed in Virginia. In his announcement, Governor McAuliffe said: “The partnership between Community Energy and Amazon Web Services, which will result in the largest solar facility in the mid-Atlantic, is indicative of the types of opportunities that my Administration is working toward through our commitment to build a new Virginia economy.”

We are excited about the positive impact that Amazon Solar Farm US East can have on the local economy in Virginia as our company continues to push for more renewable energy sources nationwide. To date, we have made significant progress toward advancing renewable energy by teaming up with developers in Indiana and North Carolina to announce new wind farms in those states as well as the new solar installation on Virginia’s Eastern Shore. These projects will deliver power to electrical grids that supply current and future AWS Cloud Computing data centers. Together these three projects will generate enough new renewable energy to power the equivalent of 122,000 US homes, or a city about the size of Pittsburgh, Pennsylvania.   We expect that the contracts we have with developers for these three installations will allow AWS to meet or exceed its goal of using 40 percent renewable power in our data centers globally by the end of 2016. That will be a major step forward in meeting our long-term goal of 100 percent renewable power for our global infrastructure.

The ground-breaking ceremony for the Amazon Wind Farm US East in North Carolina, which included state and local government officials such as North Carolina Governor Pat McCrory.

When Amazon’s solar and wind farms turn on that much clean power, over time the power grid will have a reduced reliance on the equivalent amount of fossil fuel generation somewhere else because it’s no longer needed. We believe these investments help build the scale (and the business case) for solar and wind technology manufacturers to continue to innovate and continue to lower the costs of renewable power for everyone.

However, despite the progress that companies like Amazon are making toward the greater adoption of renewable energy sources, there is still quite a distance to go in the U.S. to make renewable energy sources widely available enough to keep up with growing demand by our industry and others. In August, The Economist stated that while in “some places wind turbines and solar cells are now a cheaper way of generating electricity than fossil fuels are,” the publication also noted that “renewables, other than hydropower, still provide less than 3 percent of the world’s energy.”

That gets to the second point The Economist editorial staff made; despite more solar and wind power sources being available that are priced competitively with fossil fuels, the percentage of the world’s energy in these categories is still very limited. It doesn’t have to be that way. By putting in place thoughtful policy – to provide certainty to businesses working in the renewable energy sector and to level the playing field with other sources of energy – our industry can accelerate and expand investments in new renewable projects, which will diversify the US electrical energy generation mix. That’s not only the right policy for businesses but it’s the right policy for our long-term energy security.

As federal energy and tax reform bills are considered by the U.S. Congress, policymakers have an opportunity to achieve an “all-of-the-above” energy policy that includes oil, gas, and renewable energy by giving investors access to a corporate structure with a tax advantage that is currently only available for fossil fuel-based energy projects. Congress should enact the Master Limited Partnerships (MLP) Parity Act, a bi-partisan bill introduced this summer that would give investors in renewable energy projects the same tax benefits that investors in the fossil fuel energy industry have been enjoying for decades.

There are also other steps that federal and state policymakers can take right now to ensure that renewable energy sources will continue to be competitively priced with fossil fuels in the near-term while wind and solar prices continue to fluctuate. First, Congress should adopt the two-year extension (through 2016) of the Production Tax Credit (PTC) for wind that was included in the tax extenders bill introduced in the Senate earlier this year; second, the Investment Tax Credit (ITC) for solar should be extended beyond 2016 if possible.  At minimum, a proposed “commence construction” provision for the ITC should be included in any federal tax extender package enacted this year. This proposed change to the ITC would adjust the federal tax credit to allow projects to qualify for the ITC if construction on the project begins prior to the statutory expiration of the existing ITC on December 31, 2016, rather than the current requirement that the project is “placed in service” by ITC expiration.

In states such as Virginia that aim to create more renewable energy jobs it is time to pursue more competitive property tax rates for large-scale renewable projects, while also considering other policies to foster the construction of renewable energy projects such as a state-level investment tax credit that could eventually help replace the federal ITC.  For those states that want to continue to attract renewable energy investment and the corresponding job creation, it is important to have the right policies in place for long-term impact to achieve the economic benefits that Governor McAuliffe, Governor McCrory, and other state leaders are seeking.

Together, advancement of these policies at both the federal and state levels could provide the runway and visibility to bring about a more abundant renewable energy future.  Amazon will continue to work hard to invest in more renewable projects as AWS continues its march toward our 100 percent renewable power goal for its global infrastructure footprint.  And, we are committed to working with our business partners, government policymakers and other stakeholders to help make it happen.

— Shannon Kellogg, Director, AWS Public Policy

The Challenge
Our City on a Cloud Innovation Challenge was designed to encourage local governments to use AWS to create innovative solutions to problems faced by local governments in the United States and other countries. Each year, we recognize winners in three categories: Best Practices, Dream Big, and Partners in Innovation.

Over the years that we have been operating this challenge, we have been impressed by the amount of energy and attention that local governments devote to creating tools, apps, and data sources that provide tangible benefits to their citizens, often assisting them with information that will allow them to lead lives that are safer more productive, and better informed.

And the Envelope Please!
I am pleased to be able to reveal the winners of the 2015 City on a Cloud Challenge!

There are three winners in the Best Practices category:

• Louisville Metro Data Initiatives (Kentucky, US) – With a focus on open data and data-driven performance, the Louisville Metro Government created a series of custom sites hosted on AWS. The sites are designed to provide citizens with meaningful insights into government priorities and actions, along with the opportunity to review, analyze, and comment on government actions and processes.
• King County (Washington, US) – This county has replaced a costly and expensive tape-powered backup and disaster recovery system with a modern, cloud-powered system that combines physical and virtual cloud-powered storage appliances with Amazon Glacier and S3, with a projected savings of $1 million in the first year, and $200K thereafter.
• South Central Development and Planning Commission (Louisiana, US) -This county created MyGovernmentOnline. This site provides cloud-powered, location-oriented tracking of assets owned by government agencies of any size.

There are two winners in the Dream Big category:

• City of Chicago (Illinois, US) – The city created OpenGrid, an open source information system that supports real-time monitoring and retrieval of historical data sources.
• City of Marquette (Michigan, US) – The proposed CemeteryHost application will allow users to quickly and easy locate graves in Marquette’s historic Park Cemetery.

We would also like to recognize Trouver of Malaysia; their cross-platform collaborative tracking platform merits an honorable mention.

Finally, there are seven winners in the category of Partners in Innovation:

• Seamless Docs (US) – A quick route to a paperless government, with conversion of existing forms and PDFs to online versions, including support for electronic signatures.
• Quicket Solutions (US) – Cloud-based solutions for law enforcement to increase officer safety, boost patrol officer productivity, and increase public convenience.
• Vizalytics Technology (US) -On-the-go insights for local businesses, residents, and government agencies, powered by real-time data sets from cities, states, and local governments.
• GrupoTX (Colombia) – Improvements to government processes which lead to a better relationship between the citizens and the government.
• DataCats (Canada) – An open source project that trivializes the creation of open data infrastructure for states, regional, and local governments.
• ViewScan (Panama) – A web application that integrates GIS and transactional information to help citizens and government track urban changes, local businesses, transportation, public services, traffic, and environmental landmarks.
• Park Smart (Italy) – A platform that provides real-time visibility into the availability of parking spaces in monitored areas.

Congratulations to all of the winners, and thanks again to everyone who took the time to create and submit an entry!

— Jeff;

As we did last month, we are running a series of webinars designed to bring you up to speed on the latest AWS services & features, and to make sure that you are aware of the best ways to put them to use.

The webinars are free but “seating” is limited and you should definitely sign up ahead of time if you want to attend (all times are Pacific):

Tuesday, November 17
AWS Device Farm allows you to test your mobile apps on real devices in an automated fashion.

• Webinar: Automated Mobile App Testing with AWS Device Farm (9 – 10 AM).
• Blog Post: AWS Device Farm – Test Mobile Apps on Real Devices.

Amazon QuickSight is a fast, cloud-powered business intelligence tool. You can build visualizations, perform ad-hoc analysis, and get business insights from your data.

• Webinar: Introducing Amazon QuickSight (10:30 – 11:30 AM).
• Blog Post: Amazon QuickSight – Fast & Easy to Use Business Intelligence for Big Data at 1/10th the Cost of Traditional Solutions.

CloudWatch Dashboards lets you to build customized metrics dashboards that display Amazon CloudWatch metrics from one or more AWS regions.

• Webinar: Introducing Amazon CloudWatch Dashboards (Noon – 1 PM).
• Blog Post: CloudWatch Dashboard – Create & Use Customized Metrics Views.

Wednesday, November 18
The Amazon ECS CLI gives you command-line access to Amazon EC2 Container Service. You can create clusters, list container instances, create and manage tasks, and more.

• Webinar: Introducing Amazon ECS CLI (9 – 10 AM).
• Blog Post: EC2 Container Service Update – Container Registry, ECS CLI, AZ-Aware Scheduling, and More.

AWS Config Rules give you the ability to create compliance-checking rules that look for desirable or undesirable configurations of AWS resources.

• Webinar: Introducing Config Rules (10:30 – 11:30 AM).
• Blog Post: AWS Config Rules – Dynamic Compliance Checking for Cloud Resources.

Amazon S3’s Standard – Infrequent Access storage class is an economical way to store data that is rarely accessed.

• Webinar: Introducing a new storage class for Amazon S3: Standard-Infrequent Access  (Noon – 1 PM).
•  Blog Post: AWS Storage Update – New Lower Cost S3 Storage Option & Glacier Price Reduction.

Thursday, November 19
Over the past couple of years we have identified and refined a set of best practices for designing and building big data application that run on AWS.

• Webinar: Architectural Patterns and Best Practices for Big Data on AWS (9 – 10 AM).
• AWS Big Data Blog.

Amazon Aurora is a fast and highly scalable relational database designed to be compatible with MySQL.

• Webinar: Amazon Aurora Deep Dive (10:30 – 11:30 AM).
• Blog Post: Amazon Aurora – New Cost-Effective MySQL-Compatible Database Engine for Amazon RDS.

Amazon Redshift lets you build cost-effective data warehouses that can store well over a petabyte of data. Amazon Machine Learning helps you to perform predictive analytics on your data.

• Webinar: Advanced Analytics with Amazon Redshift and the Amazon Machine Learning Service (Noon – 1 PM).
• Blog Posts: Amazon Machine Learning – Make Data-Driven Decisions at Scale and Amazon Redshift – The New AWS Data Warehouse.

— Jeff;