The mean time to resolve (MTTR), the industry gold standard for success and efficiency, proves to be an inaccurate metric for success. The 2021 VOID report by chaos engineering company Verica set to come to conclusions about how to tackle software-based failures, but do to the distribution of data, the company found MTTR wasn’t a reliable metric.
Verica announced the launch of The Verica Open Incident Database (the VOID) to make public incident reports available to everyone in a single place.
The report spans from September 2021 back through 2008 and contains public records of all available formats. As of the time of publishing the VOID contains 1,803 reports for 598 organizations.
“Based on the distribution of the data we see in the VOID, measures of central tendency like the mean, aren’t a good representation of positively skewed data. The mean will be influenced by the skewed spread of the data, and the inherent outliers. Central tendency measures of incident duration are inherently too noisy to be measures of success for incident response,” the report stated.
Other findings include that only one-quarter of incident reports took the time to identify the root cause and less than a percent studied near miss reports.
Also, over half of incidents are externally resolved in under two hours. These findings will require a paradigm shift in thinking about how to respond to and analyze incidents, according to the company.
“We’re thrilled to announce a project that increases transparency and information sharing about software-based failures and outages. Much in the same way that airline companies set aside competitive concerns in the late 90s to improve flight safety, the tech industry has an immense body of commoditized knowledge that it could share more broadly. This project begins the process of sharing this information in order to learn from each other, and push our industry forward while making what we build safer for everyone,” said Casey Rosenthal, co-founder and CEO of Verica.
