Question

if the null hypothesis is tested at the 5% level the null hypothesis should be

Answer 1

The significance level is 0.05 (or 5%) which means that there is a 5% probability that the test will suffer a type I error by rejecting a true null hypothesis. This significance level conversely translates to a 95% level of confidence, meaning that over a series of hypothesis tests, 95% will not result in a type I error.

The significance level, in the simplest of terms, is the threshold probability of incorrectly rejecting the null hypothesis when it is in fact true.

The level of statistical significance is often expressed as the so-called p-value. Depending on the statistical test you have chosen, you will calculate a probability (i.e., the p-value) of observing your sample results (or more extreme) given that the null hypothesis is true.

For example:

The p-value such as 0.03 (i.e., p = 0.03). This means that there is a 3% chance of finding a difference as large as (or larger than) the one in our study given that the null hypothesis is true. However, we want to know whether this is "statistically significant". Typically, if there was a 5% or less chance (5 times in 100 or less) that the difference in the mean exam performance between the two teaching methods (or whatever statistic we are using) is as different as observed given the null hypothesis is true, we would reject the null hypothesis and accept the alternative hypothesis. Alternately, if the chance was greater than 5% (5 times in 100 or more), we would fail to reject the null hypothesis and would not accept the alternative hypothesis. As such, in this example where p = 0.03, we would reject the null hypothesis and accept the alternative hypothesis. We reject it because at a significance level of 0.03 (i.e., less than a 5% chance), the result we obtained could happen too frequently for us to be confident that it was the two teaching methods that had an effect on exam performance.