Formed in 2009, the Archive Team (not to be confused with the archive.org Archive-It Team) is a rogue archivist collective dedicated to saving copies of rapidly dying or deleted websites for the sake of history and digital heritage. The group is 100% composed of volunteers and interested parties, and has expanded into a large amount of related projects for saving online and digital history.
Technical talent is few and far between. The programming world faces a developer shortage, and with this shortage comes a lack of cloud know-how. By now, most organizations have become multi-cloud, yet 86% of IT professionals believe a lack of these skills will slow down cloud projects.
As more teams transition from older architectures to cloud environments, the IT skills gap could pose a severe bottleneck to software development and deployment. Cloud servers require ongoing maintenance, and managing security and compliance is challenging, even for seasoned developers. Furthermore, it’s easy to overspend in the cloud—without keen insight into vendor nuances, it becomes challenging to select the most cost-effective package for the application at hand.
I recently talked with Joe Spurrier, chief technology officer and co-founder, cloudtamer.io, to see how teams can improve their cloud strategies while navigating a short supply of IT cloud know-how. According to Spurrier, increasing automation, sharing reusable templates, centralizing best practices and maintaining agnosticism are strategies to amplify cloud results. Below, we’ll expand on these areas to see how they maximize cloud benefits while retaining a lean engineering footprint.
Automation, Automation, Automation
“Automation is key to furthering your adoption and use of the cloud,” said Spurrier. The drive to automate comes as no surprise to many DevOps and SRE professionals whose goal is to automate their job away. When automating the cloud, infrastructure-as-code (IaC) is a fundamental necessity. “If you’re getting into cloud, you should probably have some scripting skills,” said Spurrier.
Automation can significantly aid scalability when setting up additional accounts since engineers must replicate similar operations, such as server provisioning, deployment patterns, monitoring and so on. Automation is also essential for shift-left testing and to increase observability into production environments. “It’s one thing to deploy an application via automation, but another to have tests in automation—that’s the next level,” said Spurrier.
Now that so many tools are API-driven, orchestration becomes programmable. A cloud service provider’s CLIs or SDKs could also be leveraged to hook into these integrations to construct such automation.
On the flip side, too much automation for high-level privileges could backfire. “Someone could break into a data center, steal a server, disconnect it, then run off,” Spurrier explained. Thus, automation will require a higher sense of security to ensure that every change is appropriately authorized and logged.
Reusable Templating
Along with automation, documenting operational processes to be shared amongst engineering teams is vital for maximizing efficiency in a lean talent landscape. The key here is to make the process self-documenting and reusable, stresses Spurrier. This will most likely involve someone on the team creating shared libraries and repositories for other team members to leverage.
One such example is Amazon’s golden AMI pipeline. Golden AMIs are virtual machines that teams can “standardize through configuration, consistent security patching and hardening.” Golden AMIs can also enable a repeatable process for performing monitoring, logging and even distribution to business units.
Reusable patterns are important for building CI/CD systems so that when you commit code, it is automatically deployed to preferred cloud providers. In general, Spurrier recommends building shared repositories that are easy to clone, with step-by-step instructions for usage. Sample applications that demonstrate components in different languages can also aid developer usability and save time.
Adopt Centers of Cloud Excellence
Another way to encourage better adoption is by creating a center of cloud excellence (CoCE). For an enterprise, having a cloud-centric project management group can be very helpful to organize knowledge and offer support. A CoCE could centralize operations, supply engineers with pre-configured containers and help distribute access to cloud accounts.
A CoCE is a good way to start to unite disparate teams around shared best practices, said Spurrier. Depending on your organization’s needs and workloads, a CoCE will focus on varying skills, he added. In the beginning, a company may have just one center for cloud knowledge. However, as adoption grows, dedicated centers may likely emerge around specific cloud providers, too.
Retain Agnosticism
Contrary to popular belief, multi-cloud steups aren’t always about lift and shift. Business units often adopt a particular cloud to suit specific workload requirements. For example, researchers may choose GCP for its data science functions, or Microsoft shops may prefer Azure.
In a multi-cloud world, management layers could help retain a modicum of vendor neutrality. For example, take Cloud Custodian, a tool developed at Capitol One and recently contributed to the CNCF. Using Cloud Custodian, engineers can run security checks across AWS, Azure and GCP and generate metrics for reporting. Such control and insights could empower teams to automate some of the multi-cloud management burdens.
When working with multi-cloud, it’s also good to consider agnostic tooling that can work across different providers, like Terraform or Kubernetes, Spurrier said.
Templating
Cloud transformation is ongoing at most organizations. With this rise, more cloud-native processes and tools are emerging. Yet, with talent in short supply, DevOps isn’t always fully prepared. To review, we covered four ways businesses can circumnavigate this issue to get an edge:
- Increased cloud automation is a necessary response to a slimmer labor pool. Though building automation requires upfront technical skill, it pays dividends in increased efficiencies.
- Reusable libraries and templates can rapidly configure new projects and do so consistently across business units.
- Consider maintaining a center of cloud excellence to centralize policies and offer support.
- Opt for agnostic tools. This helps to retain vendor neutrality and apply consistent controls across clouds.
Of these above strategies, Spurrier emphasized the importance of creating shared templates. When used right, a templating approach can be very effective, he says. “If you have a templating model, it should be easy for DevOps teams to migrate to new features. If you’re leveraging tools that set up templates, it’s easier to figure out how much infrastructure is going to cost.”
