Logistic Regression: A Self-learning Text, Third Edition (Statistics in the Health Sciences)

VI. Multiple Testing

Modeling strategy)several

statistical tests

+

Potential for incorrect “overfitted”

model, i.e., “too many”significant

test results

“Too many”:

variable(s) found significant,

butH 0 true.

The multiple-testing problem:

Should we adjust for number

of significance tests and, if

so, how to adjust?

Statistical principle:

number of significance tests increases

Note: a = test-wise error rate

ß
a* FWER

= Pr(reject H 0 i | H 0 )

= Pr(Reject at least one H 0 i | all H 0 i true)
increases

Formula:a*¼ 1 ð 1 aÞT;

whereT¼number of independent
tests ofH 0 i,i¼1,...,T

EXAMPLE

T a*

a¼ 0 : 05 ) 1 0.05

5 0.23

10 0.40

20 0.64

The modeling strategy guidelines we have

described when one’s model contains either a

singleE(Chapters 6 and 7) or severalEs (ear-

lier in this chapter) all involve carrying out

statistical significance testing for interaction

terms as well as forE terms. Nevertheless,

performing several such tests on the same data-

set may yield an incorrect “overfitted” final

modelif “too many” test results are found to

be significant.

By “too many”, we mean that the null hypothe-

sis may actually be true for some significant

test results, so that some “significant” variables

(e.g., interaction terms) may remain in the

final model even though the corresponding

null hypotheses are true.

This raises the question as to whether or not we

should adjust our modeling strategy to account

for the number of statistical tests we perform

and, if so, how should we carry out such adjust-

ment?

A well-established statistical inference principle

is that the more statistical tests one performs,

the more likely at least one of them will reject its

null hypothesis even if all null hypotheses

are true. The parametera* shown at the left, is

often called thefamily-wise error rate(FWER),

whereas the significance levelafor an individ-

ual test is called thetest-wiseerror rate.

Mathematically, the above principle can be

expressed by the formula shown at the left.

For example, the table at the left shows that if

a¼0.05, andTranges from 1 to 5 to 10 to 20,

thena* increases from 0.05 atT¼1 to 0.64 at

T¼20.

280 8. Additional Modeling Strategy Issues