Statistical Analysis for Education and Psychology Researchers

this is evidence that the data are not consistent with the null hypothesis (the

confidence interval will also exclude zero). You reject the null hypothesis and

conclude that an alternative hypothesis is feasible. If the p-value associated with the

test statistic is greater than alpha, then it is not reasonable to reject the null hypothesis

and it remains tenable (the confidence interval will include zero).

4.8 Errors in Decision Making

Whenever we test a hypothesis we make a decision, either we make a decision to reject
the null hypothesis or not to reject it. Consider, for example, a simple treatment
effectiveness design where we have two groups, a treatment group—the reading recovery
programme and a comparison group—with no special intervention. Further assume that
the response variable of interest, reading score, is treated as a continuous variable. We
could determine the effectiveness of the programme by comparing mean reading scores
for the treatment and comparison groups. The null hypothesis we would tests is:
H 0 : μt=μc

Type I Error

Suppose the population means really do not differ. In this case the null hypothesis would
be true. If we performed a statistical test of the difference between two means, the correct
finding would be to fail to reject H 0 —we say we failed to attain significance in the
statistical test at a given probability level. We conclude that it is tenable, the population
means for the treatment and comparison groups do not differ. Put simply, the reading
recovery programme had no beneficial effects. In reaching this conclusion the decision
was that we failed to reject H 0. However, it is possible that just by chance the sample
means differed substantially. The difference between the sample means may have been
sufficiently large to lead us to reject H 0 even though the null hypothesis is tenable (μt=μc).
In this case we would have taken the wrong decision or made an error in our decision
making. If you reject the null hypothesis when you should not, that is, you conclude from
a statistical test of the sample data that population means differ, (reject this null
hypothesis) when in reality the population do not differ, you make what is called a Type
I Error.
This idea is so important in hypothesis testing that we need to spend time thinking
about it. Putting the same concept another way may help. If in reality the population
means do not differ, H 0 : μt=μc, the null hypothesis is true, then there is only one error that
could be made in these circumstances.
What is it?
The error is to incorrectly reject a true null hypothesis, and conclude wrongly from
the statistical test that the population means do differ. We make a Type I Error.
If a mistake is made it is in the decision making based on results of a statistical test of
the null hypothesis. It is not a matter of probability whether population means actually
differ—either they do or they do not (if we could access the entire population we could
determine the parameters of interest and would not need a statistical test). A Type I error:
occurs when the decision is to reject the null hypothesis when it is actually true.

Statistical analysis for education and psychology researchers 112