Introduction to Probability and Statistics for Engineers and Scientists

110 Chapter 4:Random Variables and Expectation

However, if we define the functionGby

G(p)=I(2−p)

then we see from the above that

G(p+q)=I(2−(p+q))

=I(2−p 2 −q)

=I(2−p)+I(2−q)

=G(p)+G(q)

However, it can be shown that the only (monotone) functionsGthat satisfy the foregoing
functional relationship are those of the form

G(p)=cp

for some constantc. Therefore, we must have that

I(2−p)=cp

or, lettingq= 2 −p

I(q)=−clog 2 (q)

for some positive constantc. It is traditional to letc=1 and to say that the information
is measured in units ofbits(short for binary digits).
Consider now a random variableX, which must take on one of the valuesx 1 ,...,xn
with respective probabilitiesp 1 ,...,pn. As log 2 (pi) represents the information conveyed
by the message thatXis equal toxi, it follows that the expected amount of information
that will be conveyed when the value ofXis transmitted is given by

H(X)=−

∑n

i= 1

pilog 2 (pi)

The quantity H(X) is known in information theory as theentropyof the random
variableX. ■

We can also define the expectation of a continuous random variable. Suppose thatXis
a continuous random variable with probability density functionf. Since, fordxsmall

f(x)dx≈P{x<X<x+dx}

it follows that a weighted average of all possible values ofX, with the weight given tox
equal to the probability thatXis nearx, is just the integral over allxofxf(x)dx. Hence,