Anon

Inferential Statistics 367

Law of large numbers. Let XX()^1 ==(, 1 ()()()^1 XX 211 ,,... n),XX()^2 (, 1 ()^2

XX()(),,... )

2 n

(^22) , and XX()N =(,()N XXN,,)
n
N
1 2 ... be a series of N inde-
pendent samples of size n. For each of these samples, we apply the
estimator θˆn such that we obtain N independent and identically dis-
tributed as θˆn random variables θθˆnn(1)(2),ˆ ,,...θˆn()N. Further, let E(θˆn)
denote the expected value of θˆn and θθˆ(1)(2)nn,ˆ ,,...θˆ()nN. Because they
are identically distributed, it holds that^2
(^) ∑θ=θ
N =
plim E
(^1) ˆ
nk (ˆ )
k
N
() n
1

(C.4)

The law of large numbers given by equation (C.4) states that the average
over all estimates obtained from the different samples (i.e., their sample mean)
will eventually approach their expected value or population mean. According
to equation (C.2), large deviations from E(θˆn) will become ever less likely the
more samples we draw. So, we can say with a high probability that if N is
large, the sample mean

1 N∑k=θˆnk

N ()

1

will be near its expected value. This is a valuable property since when we
have drawn many samples, we can assert that it will be highly unlikely that
the average of the observed estimates such as

1 Nx∑k= k

N

1

for example, will be a realization of some distribution with very remote
parameter EX()=μ.
An important aspect of the convergence in probability becomes obvious
now. Even if the expected value of θˆn is not equal to θ (i.e., θˆn is biased for
finite sample lengths n), it can still be that plimθ=ˆn θ. That is, the expected
value E(θˆn) may gradually become closer to and eventually indistinguish-
able from θ, as the sample size n increases. To account for these and all
unbiased estimators, we introduce the next definition.

(^2) Formally, equation (C.4) is referred to as the weak law of large numbers. Moreover,
for the law to hold, we need to assure that the θˆn()k have identical finite variance.
Then by virtue of the Chebychev inequality we can derive equation (C.4). Chebyshev’s
inequality states that at least a certain amount of data should fall within a stated num-
ber of standard deviations from the mean.