Statistical Analysis for Education and Psychology Researchers

and the veracity of any generalizations. A good research report would provide
information on all important aspects of the study design and data collection procedures.
An informed reader can then judge whether or not the results claimed are reasonable.
Similarly, the validity of experimental conclusions will need to be judged against the
extent to which results are attributed solely to the causes identified in the experiment.
Again the laws of chance are involved. They allow the outcomes of an experiment to be
explained by taking into account what outcomes would be expected by the laws of chance
alone. Observation of differences among treatment groups which are too large to be
attributable to chance alone are said to be statistically significant. Data which is
generated from an experiment is the product of an experimental design. The quality of
that design, the adequacy and accuracy of data measurement and recording procedures,
the effectiveness of randomization of subjects to treatment groups, and the choice of the
number of subjects or treatments will influence the validity of experimental results.
The most important overall message to be gleaned thus far is that consideration of
statistical principles is crucial to good research design and that statistical ideas are
involved as much in data collection as in data analysis. Any data analysis will be
influenced by the method of data collection. For example, if data are collected in a
haphazard way then it may not be worth spending any time on data analysis. Clarity of
purpose and statistical awareness at the design stage of a study should lead to the
avoidance of typical problems such as small numbers of observations, missing responses
for key variables, large sampling errors, and unbalanced experimental designs. These
problems make any subsequent analysis and interpretation more difficult and possibly
less powerful. Do not leave consideration of data analysis until after data has been
collected. It may be too late at this stage to rectify a fundamental design problem which
would render impossible the answering of the research question you had in mind.

1.2 Surveys

Many studies are labelled as surveys, ranging from Charles Booth’s classic poverty
surveys of the working-class population of London (Booth, 1889–1902) to modern day
Gallup Polls, Government Surveys (General Household Survey, Labour Force Survey,
Family Expenditure Survey, British Household Panel Study, Child Development Study,
Youth Cohort Study) and opinion and market research surveys. Surveys are usually
designed to collect data from a population of interest. A population in a statistical sense
refers to a complete set of subjects, values or events that have some common
characteristic. In many survey designs the population of interest—all those to whom you
would like to generalize your findings—may be indeterminable in number or impossible
to enumerate. In this sense the term population is a theoretical concept because all
members could never be observed or counted. Most surveys involve the selection of
samples, unless they are censuses or total population surveys. A sample is usually a
collection of subjects, values or events which is finite in size, therefore quantifiable, and
represents a subgroup of a population. The idea of sampling is to use characteristics of a
selected sample to infer properties of the population from which the sample was drawn.
Rarely is it feasible or necessary to include the total population of interest in a study.

Statistical analysis for education and psychology researchers 6