Whereas survey methods vary, for example, postal self-completion, class administered
questionnaires, telephone, interview and observational surveys, the one feature most are
likely to share is the need to obtain a sample. A survey sample is usually intended to
represent the population from which it is drawn. If the sample is faulty or is not designed
to be representative, then it is not reasonable to generalize beyond the achieved sample.
This presents the researcher with a problem of external validity or generalizability. The
ability to generalize findings relates to the definition of the population of interest, the
sampling method used and the validity and reliability of measurement.
Sources of variabilityA good sample design will minimize the amount of variability in observations or
measurements to an acceptable level given the purpose and required precision of a survey
or experiment. Variability is inherent in measurements on subjects. For example, consider
a teacher who selects a sample of school children and measures for each child his or her
assertiveness behaviour using the School Motivation Analysis Test (SMAT instrument
described by Boyle and Houndoulesi, 1993) and then estimates an average assertiveness
score. If on the following day the teacher repeats the testing, it is very likely that the two
measurements for each child will vary as will the two averages of the first and second set
of measurements. Such variability may be due to random variation in measurement, a real
change in the children’s assertiveness (a dynamic trait) or a combination of both.
In designing a survey to estimate teenagers’ assertiveness, for example, consideration
should be given to three potential sources of variability or error. The full inventory has
190 items, and some students may become bored or tired and not answer all items. This
would introduce a measurement error which is an example of nonsampling bias. If the
teacher had selected a non-probability sample that is a non-random sample, then the
sample may not be representative of all teenagers (not every teenager would have an
equal or known chance of being selected). This would introduce a selection bias and is
an example of sampling bias. Finally any sample is subject to sampling error or
sampling variability. Put simply this means that any particular sample average, given
certain assumptions, will vary from another independent sample average. This idea of a
distribution of sample averages and how it relates to sampling error will be explored in
Chapter 4. The implication for planning is that the precision of a sample statistic such as
an average assertiveness score is related to sampling error. In fact the precision of a
sample statistic decreases as sampling error increases. Sampling error is influenced by
sample size and variability of what is being measured in the population, in this case
assertive-ness. In this example, the smaller the population variability in assertiveness
(more homogeneous) the smaller will be the sampling error; this will provide a more
precise average assertiveness score. Larger sample sizes also reduce sampling error,
which is one reason why larger samples are preferable to smaller ones. ‘Small’ in survey
and experimental research is generally taken to mean less than 30 subjects, but this is
only a guideline. Generally for both experiments and surveys more subjects are better
than fewer subjects up to a point. The more subjects that participate in an experiment, the
more it is likely that randomization to treatment groups will be effective. Consequently,
on average groups will be similar because any individual differences among subjects will
be averaged out by the random allocation of subjects to groups.
Statistical analysis for education and psychology researchers 4