AIOps takes center stage to topple barriers as modern data application growth accelerates
There seems to be a chasm between DevOps and big data teams in the enterprise, and the bridge over that chasm is long and treacherous.
In fact, data analysts who are charged with managing and making sense of the growing volumes of data in most organizations traditionally have been excluded from the strategy-making and operations functions of DevOps personnel. Yet, data analysts are critical to the success of software development and at every stage of the software delivery pipeline.
DevOps Under Siege as Modern Data Applications Proliferate
Let’s take a deeper look. We know that DevOps personnel are on the front lines in ensuring that all systems, both new and existing, perform as expected in a production environment. But, even today, DevOps teams struggle in dealing with technical challenges when faced with complex data pipelines and the underlying big data stack that supports them, including:
- Root cause analysis — determining if an application issue is due to application code, config errors or resource contention.
- Correcting inefficiency in code.
- Improperly configured auto-scale clusters.
- Lack of actionable insight into how clusters were tuned at both a cluster and job level.
- Difficult access to support engineers and promotion of DevOps collaboration.
And these are just the tip of the iceberg.
The pressure is on. On average, companies with data-driven environments garner up to 50 percent higher market value than their competitors, according to IDC, and big data is a contributing factor. Thus, big data implementations have become a top priority for businesses of all sizes, and on multiple fronts: revenue acceleration, creation of new products and contributing to better, quicker decision-making. But the unfortunate truth is that, as big data projects move from pilot to production, they invariably encounter serious performance problems that frustrate both data analysts and IT operations.
What’s more, DevOps and big data personnel face a mutual problem: lack of centralized, top-down visibility into their end to end data pipelines and applications. And, as a result, they face challenges in pinpointing the source of problems—namely, that problems can become visible anywhere in the stack, but their root cause could be elsewhere in in the stack. And, these problems result from root causes as diverse as bad code, inefficient data partitioning, mismatched system configuration settings, best-guess resource allocation and infrastructure issues. Furthermore, a big data stack running multiple applications faces resource contention and prioritization issues that can reduce both individual application performance and overall system performance. And streaming applications create their own challenges;: for example, if a streaming application starts to lag in processing data in real-time, diagnosing the root cause takes considerable time and effort.
The industry is faced with a reality where the problems are outrunning the solutions. In the case at hand, organizations have relied on brute force and expensive custom solutions for making analytics-based recommendations more user-friendly to improve individual application and full-stack performance.
AIOps Takes Aim
AIOps, or artificial intelligence for IT operations, is gaining steam in the enterprise, with many new use cases appearing as data volumes grow and systems become more complicated. Gartner expects AI to pervade every aspect of managing IT, including application performance management (APM), IT infrastructure monitoring, cloud platform monitoring, performance management, capacity management, the delivery tool chain (CI/CD) and IT service management.
In fact, AIOps is on a tear: Just three years ago, it was only a concept. Today, Gartner predicts that by 2019, 25 percent of global enterprises will have implemented an AIOps platform to support two or more major IT operations.
But even as it continues to evolve, AIOps today can deliver automated recommendations to solve expensive, challenging and time-consuming problems with big data deployments and across entire complex stacks. Without AIOps, there would be simply too many variables to correlate manually.
AIOps makes APM for modern data application possible. It does so by helping to eliminate painstaking, trial-and-error configuration, troubleshooting and remediation workflows that have long plagued big data operations teams. And, for the first time ever, dataOps and DevOps teams can troubleshoot, support SLAs, remediate issues and manage large-scale deployments in an agile and dependable manner.
It’s a new environment for the modern data center. As part of an APM solution, AIOps is hard at work helping automate and guarantee the performance of big data applications and addressing the primary challenges that DevOps faces with the unrestrained growth of big data.
It could still be a bumpy ride—organizations of all kinds face unprecedented increases in data volumes. But now, working together, DevOps and AIOps are better-equipped to contend with the challenges.
