Question

• Imagine that a friend of yours is pleased that the results of a Kruskal–Wallis test your friend ran as part of a dissertation turned out to be statistically significant at alpha = .05. Your friend followed up this significant result with 15 Mann–Whitney tests and found out that the results were also statistically significant at alpha = .05. Suggest one (1) overall strategy to your friend to interpret your friend's finding from the tests. Provide a rationale for your suggestion.

Answer 1

If we run a Kruskal wallis test at alpha = 0.05, we mean that the probability of occuring a type 1 error is 5%. If we run multiple Mann Whitney tests, the probability for a type 1 error increases. Hence, we should apply a Bonferroni correction to the significance level used.

If m pairwise tests are done in total, then the significance level used for each test should be alpha/m. Here, m = 15 and alpha = 0.05, therefore each test should be checked for significance at the level of 0.05/15 = 0.0033. The rationale for the suggestion is that if we take a 5% probability of type 1 error for each test, overall we are increasing the likelihood if we perform multiple such tests. Hence, on an average, the significance level used should be alpha/m.