I was asked the other day what advantage a 5×5 risk matrix has over a 4×5 and it started me thinking. A risk matrix is meant to help categorize, prioritize and compare risk, so what difference do 5 or 4 rows or columns really make? I assumed it would depend on how precisely the severity and likelihood ranges had been defined but what I found out was so much more. I have always been mildly skeptical of risk matrices, struggling to see how all that risk data can be condensed and simplified into a single box, and still remain meaningful. I was delighted to find my skepticism was well placed!
The common factor of risk matrices
My research highlighted that there is no scientific method for designing the scale used in a risk matrix. From the numerous and varied scales I have encountered in aviation and elsewhere, the common factor is they are typically ordinal scales. An ordinal scale has no fixed distance between the levels; the numbers represent a rank position. Questions with subjective responses are often ordinal, for example, “how much pain are you in?” could be answered with “none”, “a little”, “some”, “a lot”, “excruciating”. The responses go from least to most pain, but it’s not clear whether the difference between “none” and “a little” is bigger, smaller, or the same as the difference between “a lot” and “excruciating”. This also emphasizes the subjective nature of the scale. What’s excruciating to me maybe merely “a little painful” to you.
Ordinal responses may be transformed in any way that preserves their order, which in a 5×5 risk matrix could be 1-5 or even 0,5,37,40 and 103. The numbers are irrelevant as long as the order stays the same.
Using the previous example, we cannot say that “excruciating” is twice as painful as “some”. Similarly, 70 degrees is not twice as hot as 35 degrees, because multiplication cannot be applied to an ordinal scale. However, this is what appears to have been done in the CAA UK’s CAP 795, Safety Management Systems (SMS) guidance for organizations (Table 1). The numbers imply certain comparisons between risks, suggesting that Remote/hazardous is twice as risky as improbable/major; the result of committing the mathematical no-no of multiplying an ordinal scale.