Statistical Analysis for Education and Psychology Researchers

Significance Tests and Confidence Intervals

Significance test for overall model fit

For a simple linear regression model with one explanatory variable and ten cases, the
sources of variation (sums of squares, SS) and degrees of freedom can be partitioned as
follows:
y= β 0 + β 1 x 1 + ε
SSY SSmodel SSe
Total Model sums of squares Error sums of squares
df=n−1 df for model=(number of parameters in model−1) df=n−(number of b parameters)
9 (2–1)=1 8
Mean square model=SSmodel/dfmodel Means square error=SSe/dferror

The model sums of squares, SSmodel is evaluated by subtraction of the error sums of
squares from the total corrected sums of squares. That is SSmodel=(SSYY−SSe)
=(1030.4−202.636)=827.764.
The MSmodel term is then evaluated as=827.764/1=827.764 (associated df=1). The
MSerror term is 202.636/8=25.329 (associated df=8).
If the model makes a significant contribution to the prediction of the response variable
then SSmodel should be greater than SSe. This is, in fact, equivalent to the alternative
hypothesis when expressed in terms of the mean squares, H 1 : MSmodel> MSerror>1.
The null hypothesis tested is, H 0 : MSmodel=MSerror=1. The test statistic used is F. This
null hypothesis is a test of the strength of the linear relationship between the explanatory
variable(s) and the response variable. An F statistic close to 1 would indicate that the null
hypothesis of no linear relationship is true. As the F statistic increases in size this
provides evidence for the alternative hypothesis that there is a linear relationship between
the response and explanatory variables, i.e., rejection of the null hypothesis. In this
example, F=MSmodel/MSerror=827.764/ 25.329=32.680 with 1 and 8 df.

Interpretation

To evaluate the statistical significance of the obtained F-statistic we refer to Table 7,
Appendix A4, Critical values of the F-distribution. If we select a 5 per cent significance
level (there is one table for each level of significance) and want to look up the critical F-
value using this table we need to know the two degrees of freedom which are associated
with our observed F-statistic. In this example the values with which we enter the table are
1 (numerator) and 8 (denominator). In Table 7, the numerator df are shown along the top
row and the denominator df are shown down the left column. We therefore locate 1 on
the top row and follow the column down until we intersect with 8 for the denominator df.
The critical value of F entered in the table is 5.32. Since the obtained value of F is larger
than the tabled critical value, 32.68>5.32, we can reject the null hypothesis and conclude
that the independent variables, just one in this case, makes a significant contribution to
the prediction of the response variable.

Statistical analysis for education and psychology researchers 264