Multiple-testing literature:
researcher knows in advance
number of tests
Modeling strategy (“best model”)
problem: researcher does not
know in advance number of tests
Bonferroni-type adjustment not
possible when determining a
“best model”
(Cannot specifyTin advance)
Ad hoc procedure:
Drop all variables in a
nonsignificant chunk, e.g., all
interaction terms
(Drawback: BW elimination may
find significant effects
overlooked by chunk test)
Summary about multiple testing:
No full-proof method avail-
able.
Finally, the literature on multiple-testing
focuses on the situation in which the
researcher knows in advance how many tests
are to be performed. This is not the situation
being addressed when carrying out a modeling
strategy to determine a “best” model, since the
number of tests, say for interaction terms, is
only determined during the process of obtain-
ing one’s final model.Consequently, when determining a “best”
model, a Bonferroni-type adjustment is not
possible since the number of tests (T)tobe
performed cannot be specified in advance.One approach for reducing the number of
tests, nevertheless, is to use the results of non
significant chunk tests to drop all the variables
in the chunk, rather than continue with back-
ward elimination (using more tests). However,
note that the latter may detect significant
(interaction) effects that might be overlooked
when only using a chunk test.Thus, in summary, there is no full-proof
method for adjusting for multiple-testing
when determining a best model. It is up to the
researcher to do anything, if at all.282 8. Additional Modeling Strategy Issues