Formed in 2009, the Archive Team (not to be confused with the archive.org Archive-It Team) is a rogue archivist collective dedicated to saving copies of rapidly dying or deleted websites for the sake of history and digital heritage. The group is 100% composed of volunteers and interested parties, and has expanded into a large amount of related projects for saving online and digital history.
Docker’s strengthening its infrastructure components to accommodate the volume of use and wide range of use cases that inevitably crop up from such hockey-stick growth. The latest improvements come by way of refinements made to Docker Hub over the last month, according to an announcement made by the company yesterday.
The Docker Hub team reported that many of the changes were precipitated by performance issues experienced at the end of 2014, following the addition of security enhancements to prevent brute force login attempts. They reported that the enhancements added “overhead to the login process.”
“Under normal load there wasn’t much difference, but as the number of login attempts increased the site performance started to degrade,” they wrote. “The decreased performance affected not just the Docker Engine logins, push, and pulls, but also the Docker Hub website as a whole.”
In the process of identifying immediate issues causing the problem, Docker’s engineers audited the entire system and is in the middle of rearchitecting the authorization, push and pull system to improve performance and isolate systems for a bulkhead effect, ensuring that there aren’t cascading problems.
In the immediate term, some of the fixes already made include bulking out infrastructure capacity, including a tripling of servers in the Docker cluster for four-times the compute power and a production database upgrade for twenty times the RAM. It also improved login infrastructure to include added database indexes and more caching. The team also tuned queries for speed and fixed an issue that was causing database locks. Docker also switch its background worker queue broker from Redis to RabbitMQ and moved background workers to their own server cluster. Additionally, improvements were made to monitoring and metrics gathering, as well as to search functions.
Meanwhile, down the line they expect to make further improvements that include API throttling, breaking up larger services to improve scalability, replacing the brute force prevention library with a better performing library and adding API metrics and monitoring to head off performance issues earlier.
“We have made some good progress, but we are not finished yet, and we still have a lot more planned,” the Docker Hub team wrote. “We will continue to make improvements with every release, so that Docker Hub only gets faster and more reliable as time goes on.”
Such hiccups as the ones in December are understandable given the meteoric growth of the Docker community. Even just six months ago, the total number of downloads of Docker were under 3 million. By December that number had risen to 100 million. In spite of some questions about the security of Docker containers, Docker’s trajectory seems promising. Just in September the company closed on $40 million in series C investment from Sequoia and over the past few months it has been announcing a number of high-powered partnerships, including those with VMware, Microsoft and AWS.
Docker’s most recent partnership was announced last week when it teamed up with Google so that the online giant could offer a private registry service for Docker on its cloud platform. Currently in beta, the Google Container Registry will provide Google cloud customers with secure hosting, sharing and management of private container repositories that will include strong access control and server-side encryption.
