Introduction to Probability and Statistics for Engineers and Scientists

4.4Expectation 109

repetitions of the experiment. That is, if we continually roll a fair die, then after a large
number of rolls the average of all the outcomes will be approximately 7/2. (The interested
reader should try this as an experiment.) ■

EXAMPLE 4.4b IfIis an indicator random variable for the eventA, that is, if

I=

{

1ifAoccurs

0ifAdoes not occur

then

E[I]= 1 P(A)+ 0 P(Ac)=P(A)

Hence, the expectation of the indicator random variable for the eventAis just the
probability thatAoccurs. ■

EXAMPLE 4.4c EntropyFor a given random variableX, how much information is conveyed
in the message thatX =x? Let us begin our attempts at quantifying this statement by
agreeing that the amount of information in the message thatX=xshould depend on
how likely it was thatX would equalx. In addition, it seems reasonable that the more
unlikely it was thatX would equalx, the more informative would be the message. For
instance, ifXrepresents the sum of two fair dice, then there seems to be more information
in the message thatXequals 12 than there would be in the message thatXequals 7, since
the former event has probability 361 and the latter^16.
Let us denote byI(p) the amount of information contained in the message that an event,
whose probability isp, has occurred. ClearlyI(p) should be a nonnegative, decreasing
function ofp. To determine its form, letXandYbe independent random variables, and
suppose thatP{X=x}=pandP{Y=y}=q. How much information is contained in
the message thatXequalsxandYequalsy? To answer this, note first that the amount
of information in the statement thatXequalsxisI(p). Also, since knowledge of the fact
thatXis equal toxdoes not affect the probability thatYwill equaly(sinceXandYare
independent), it seems reasonable that the additional amount of information contained in
the statement thatY=yshould equalI(q). Thus, it seems that the amount of information
in the message thatXequalsxandYequalsyisI(p)+I(q). On the other hand, however,
we have that

P{X=x,Y=y}=P{X=x}P{Y=y}=pq

which implies that the amount of information in the message thatXequalsxandYequals
yisI(pq). Therefore, it seems that the functionIshould satisfy the identity

I(pq)=I(p)+I(q)