Formed in 2009, the Archive Team (not to be confused with the archive.org Archive-It Team) is a rogue archivist collective dedicated to saving copies of rapidly dying or deleted websites for the sake of history and digital heritage. The group is 100% composed of volunteers and interested parties, and has expanded into a large amount of related projects for saving online and digital history.
For all the technological advancements we are witnessing in AI and machine learning, the practical benefits remain elusive in the non-tech business world, frustrating decision-makers as AI investment climbs. For example, over the past several years, many Fortune 500 companies have invested in strong data science teams and AI labs, but still struggle when it comes to producing and scaling their models. Moreover, a recent report by Algorithmia found that, while enterprise budgets for AI and ML are rapidly increasing for 83% of organizations studied, deployment challenges persist.
These deployment issues often begin when leadership fails to exert enough upfront engineering focus to ensure their proof of concepts (POCs) are “real-world ready.” The challenges are compounded when businesses put these models into action using insufficient, outdated or cobbled-together IT infrastructures. This combination results in POCs that either fall apart under real-world conditions or are rife with inefficiencies. Fortunately, cloud tech today provides the opportunity to overcome these challenges.
Businesses can enable development, experimentation and scaling of data science and AI solutions with more flexibility and speed than ever through cloud platforms. A design-led AI project that uses the tools available in the cloud can help organizations create business impact and realize ROI much faster. So while the role of data scientist may have been the ‘sexiest job’ in recent years, we may be seeing a new ‘sexiest job’ title emerging: the cloud AI engineer.
Why do we need a Cloud AI Engineer?
Traditionally, a successful AI project in the cloud requires a team with different skills and titles, including data scientists, big data engineers, cloud engineers and full stack developers. This has led businesses to look at ways to merge these specialties into one specific title. Enter the cloud AI engineer who, theoretically, would be able to cover all these duties — including knowledge of and experience with the tools and accelerators that cloud platforms now offer, as well as demonstrated aptitude for fast-paced development in AutoML. These are skills and tasks previously spread out across other areas and roles within the organization.
What is the Job Description?
So what would this job actually look like? The cloud AI engineers of the future will focus on the deployment of AI and ML models at scale in the cloud, and on integrating them with existing products and IT systems. Additionally, cloud AI engineers would need knowledge of the different types of machine-learning algorithms; familiarity with frameworks like Tensorflow and PyTorch; hands-on experience with data processing; analysis and feature selection using analytic tools and ETL/ELT frameworks (Apache Spark, Hive, Amazon, Athena, etc.); and model deployment experience on cloud platforms like AWS, Azure or Google Cloud.
Obviously, tackling all of these would be a huge undertaking for one individual; given the vast workloads on both the cloud and engineering fronts, it isn’t possible to replace these workers and their skills with just a handful of cloud AI engineers. Instead, cloud AI engineers will sit between the two disciplines and the workforces and will serve as a key bridge to easily deploy models and integrate them with enterprise systems and tools for consumption.
Where are We Going to Find Cloud AI Engineers?
Given that cloud AI engineers will have a robust level of responsibility, they will quickly become some of the most sought-after team members — but that also means talent for the role will likely be in short supply.
Additionally, individuals with this broad skill set will not appear overnight — at least not en masse. Therefore, to build the cloud architecture capable of delivering the real-world AI results that businesses need today, tech companies need to find ways to break down the internal divides between data scientists and cloud engineers so that they can cross-train, learn and develop skills in each area. If this is done correctly, not only will tech companies have a more well-rounded workforce, but they will be able to deliver the practical AI success that non-tech businesses are looking for today.
Despite excitement about AI and its benefits, operational challenges remain. By focusing on the cloud and building up specialists in this space, AI can begin to drive the practical results the business world needs far more quickly.
