Introduction to Probability and Statistics for Engineers and Scientists

230 Chapter 7: Parameter Estimation

In the optional Section 7.8, we consider the problem of determining an estimate of an
unknown parameter when there is some prior information available. This is theBayesian
approach, which supposes that prior to observing the data, information aboutθis always
available to the decision maker, and that this information can be expressed in terms of
a probability distribution onθ. In such a situation, we show how to compute theBayes
estimator, which is the estimator whose expected squared distance fromθis minimal.

7.2Maximum Likelihood Estimators

Any statistic used to estimate the value of an unknown parameterθis called anestimator
ofθ. The observed value of the estimator is called theestimate. For instance, as we shall
see, the usual estimator of the mean of a normal population, based on a sampleX 1 ,...,Xn
from that population, is the sample meanX =

∑
iXi/n. If a sample of size 3 yields the
dataX 1 =2,X 2 =3,X 3 =4, then the estimate of the population mean, resulting from
the estimatorX, is the value 3.
Suppose that the random variablesX 1 ,...,Xn, whose joint distribution is assumed
given except for an unknown parameterθ, are to be observed. The problem of interest
is to use the observed values to estimateθ. For example, theXi’s might be independent,
exponential random variables each having the same unknown meanθ. In this case, the
joint density function of the random variables would be given by

f(x 1 ,x 2 ,...,xn)

=fX 1 (x 1 )fX 2 (x 2 )···fXn(xn)

=

1

θ

e−x^1 /θ

1

θ

e−x^2 /θ···

1

θ

e−xn/θ,0<xi<∞,i=1,...,n

=

1

θn

exp

{

−

∑n

1

xi/θ

}

,0<xi<∞,i=1,...,n

and the objective would be to estimateθfrom the observed dataX 1 ,X 2 ,...,Xn.
A particular type of estimator, known as themaximum likelihoodestimator, is widely
used in statistics. It is obtained by reasoning as follows. Letf(x 1 ,...,xn|θ) denote the joint
probability mass function of the random variablesX 1 ,X 2 ,...,Xnwhen they are discrete,
and let it be their joint probability density function when they are jointly continuous
random variables. Becauseθis assumed unknown, we also writef as a function ofθ.
Now sincef(x 1 ,...,xn|θ) represents the likelihood that the valuesx 1 ,x 2 ,...,xnwill be
observed whenθis the true value of the parameter, it would seem that a reasonable estimate
ofθwould be that value yielding the largest likelihood of the observed values. In other
words, the maximum likelihood estimateθˆis defined to be that value ofθmaximizing
f(x 1 ,...,xn|θ) wherex 1 ,...,xnare the observed values. The functionf(x 1 ,...,xn|θ)is
often referred to as thelikelihoodfunction ofθ.