Harnessing the Value of Log Data Analytics

Each day, the average enterprise’s cloud applications, containers, compute nodes and other components throw off millions of tiny logs. Each log is a file whose data describes an event such as a user action, service request, application task or compute error. Logs also capture messages that applications and other components send to one another.

There’s a wealth of information and value hidden in those logs, which is why businesses are so keen on using them to address business needs like customer engagement, IT security and cloud operations. 

The need for understanding and managing this data is so high that Markets and Markets valued the market for log analytics data at $1.9 billion in 2020. Analysts predict this to grow to $3.7 billion by 2025. It’s not just the sheer growth of data driving this market—it’s also the demand to gain business insights. 

That’s why ChaosSearch recently conducted a survey to better understand how businesses are currently using and managing their log data. Insights were collected from practitioners at 50 medium-to-large enterprise organizations. What the survey found is that organizations see a lot of value from tapping into their log data to achieve better outcomes across both business and IT functions. That said, these practitioners face challenges handling the scale requirements and they know there are still more opportunities for them to expand the scope and discipline for log management.

Top Use Cases

This study confirmed that businesses are using log data in a number of ways, spanning both business and IT functions. We see that more than 90% of respondents are using log management and analytics for at least two primary use cases. It’s not surprising that the top use cases for log management are security (70%) and IT monitoring (68%), but many are prioritizing BI & analytics (46%) and business operations (28%) as well.  

It’s interesting to look more closely at the security use cases: Investigations (45%) and insider threats (45%) top the list of SecOps uses for log data. Audits and compliance (32%), threat hunting (32%) and anomaly detection (30%) also made the list. 

Growing Desire to Support Business Operations

We also saw that these practitioners want to use log data to support business operations—specifically to drive better business outcomes and improve competitiveness. For example:

1. A global industrial equipment manufacturer attached IoT devices to farm equipment to track weather, temperature, humidity, time of day, depth of planting, soil conditions, seeds planted and geo-routing on farms. This raw data is continually collected and analyzed within the central log analytics solution, thereby allowing the vendor to deliver farm productivity reporting and recommendations back to their clients.
2. A biopharmaceutical firm tracks cell culture logs with alerts on temperature and humidity. The company uses its log analytics platform to store long-term data to run queries that help in hypothesis testing. They also conduct various long-tail analyses that reveal trends and allow scientists to make informed decisions about future experiments. 
3. A large online retailer leverages log data from web visits and other customer activities to develop 360-degree profiles of customers and analyze the steps in the “customer journey” to identify opportunities for improved efficiency and increased revenue capture.

Tackling Challenges of Growth and Complexity

Even with more interest in using log data for security and business operations, the surveyed practitioners called out a range of challenges. The top three being: Complex infrastructure (48%), managing costs (42%) and managing growth (34%). They also said the lack of centralization, reporting and data quality were problems. These challenges are perhaps exacerbated by the expectation that data will continue to grow and data retention will increase. 

There are a lot of variables that drive the volume of log data generation, like the number of users, devices, applications, IT environments and infrastructure elements. The survey found two things were universal—data growth rates are high and are a source of pain and data retention is an increasing priority. 

Among the participants, 94% ingest at least one TB or more of data per day, with 18% in the 10TB + range. Consider this—the average daily ingest volume for these companies was 7.9 TB! 

The problem is, despite the massive amounts of data being ingested daily, only 28% of respondents are capturing 80% of log data or more today. And 78% believe that capturing more than 80% of log data is ideal. That’s a big gap. 

Practitioners Rate Most Important Log Management Capabilities

We also learned that these practitioners are generally satisfied with their existing solutions, but many of them still expect to bring in a new solution for log management and analytics within the next three years. That’s because they want better scale and a cloud-native solution. 

When we asked about important capabilities for a log management platform, scalability (58%), resilience (54%) and advanced analytics (52%) topped the list. 

Budgets Increase to Meet Demand for Log Analytics

To address these challenges and opportunities before them, survey respondents are growing their budgets; 86% anticipate their budgets will grow by at least 20% and 36% expect an increase of 50% or more. These budgets will not only support more data capture and increase retention periods but also help them go after the growing use cases for supporting the business. In fact, 68% said they expect the number of use cases to grow in the next 12 months.

There is no doubt: Companies who value log data and invest in capturing data will harness insights that will propel their businesses forward.