Pepperdata’s inaugural Big Data Performance Report 2020 revealed that enormous waste exists within enterprise data applications that are not optimized by solutions that allow for observability and continuous tuning. There is also tremendous potential to optimize and reduce that waste.
This is happening because as more complex big data applications migrate, the likelihood of resource misallocation rises. Over the next two years, the public cloud services market is expected to grow from $266.4 billion to $350 billion, according to Gartner.
“When we analyzed the data, we were amazed to see how much underutilization and other wasted resources there were—unnecessarily driving costs up,” said Joel Stewart, the VP of customer success at Pepperdata. “The failure to optimize means companies are leaving a tremendous amount of money on the table—funds that could be reinvested in the business or drop straight to the bottom line. Unfortunately, many companies just don’t have the visibility they need to recapture the waste and increase utilization.”
The highest amount of net wastage was found in Spark clusters and jobs that are dominating spend across clusters.
Job failures cause serious performance degradation, and consume significant computational resources. Pepperdata found that some clusters will fail above 10%, and Spark applications tend to fail more often than MapReduce.
Comparing jobs used and wasted, the average wastage across 40 large clusters is 60+%. Typically, within 95% of jobs, there is little wastage and Major wastage was found in 5% to 10% of total jobs, according to Pepperdata.
The research showed that cloud optimization delivered big savings. 25% of users are able to save a minimum of $400,000 per year. At the higher end, the most successful users are able to save a projected $7.9 million for the year.
Also, three quarters of customer clusters immediately win back task hours with the visibility offered by real cloud optimization. Most enterprises are able to increase task hours by a minimum of 14%. Some enterprises are able to increase task hours by as much as 52%, the research found.
Additional details on the findings are available here.
