176 Chapter 6. Entropy, temperature, and free energy[[Student version, January 17, 2003]]
Shannon’s formula also has the reasonable property that the disorder of a random message is
maximum when every letter is equally probable. Let’s prove this important fact. We must maximize
Iover thePi,subject to the constraint that they must all add up to 1 (the normalization condition,
Equation 3.2 on page 66). To implement this constraint, we replaceP 1 by 1−
∑M
i=2Piand maximize
overall the remainingPis:
−I/(NK)=[
P 1 lnP 1]
+
[∑M
i=2PilnPi]
=
[(
1 −
∑M
i=2Pi)
ln(
1 −
∑M
i=2Pi)]
+
[∑M
i=2PilnPi]
Set the derivative with respect toPjequal to zero, using the fact thatddx(xlnx)=(lnx)+1:
0=
d
dPj(
−
I
NK
)
=
[
−ln(
1 −
∑M
i=2Pi)
− 1
]
+
[
lnPj+1]
Exponentiating this formula gives
Pj=1−∑M
i=2Pi.But the right hand side is always equal toP 1 ,soall thePjare the same. Thus the disorder is
maximal when every letter is equally probable, and then it’s just given by Equation 6.1.
T 2 Section 6.1′on page 205 shows how to obtain the last result using the method of Lagrange
multipliers.
6.2 Entropy
6.2.1 The Statistical Postulate
What has any of this got to do with physicsorbiology? It’s time to start thinking not of abstract
strings of data, but of the string formed by repeatedly examining the detailed state (ormicrostate)
of a physical system. For example, in an ideal gas, the microstate consists of the position and speed
of every molecule in the system. Such a measurement is impossible in practice. But imagining what
we’d get if wecoulddo it will lead us to the entropy, whichcanbedefined macroscopically.
We’ll define the disorder of the physical system as the disorder per observation of the stream of
successively measured microstates.
Suppose we have a box of volumeV,about which we know absolutely nothing except that it is
isolated and containsNideal gas molecules with total energyE. “Isolated” means that the box
is thermally insulated and closed; no heat, light, or particles enter or leave it, and it does no work
on its surroundings. Thus the box willalwayshaveNmolecules and energyE.What can we say
about the precise states of the molecules in the box, for example their individual velocities? Of
course the answer is, “not much”: The microstate changes at a dizzying rate (with every molecular
collision). We can’t say a priori that any one microstate is more probable than any other.
Accordingly, this chapter will begin to explore the simple idea that after an isolated system has
had a chance to come to equilibrium, the actual sequence of microstates we’d measure (if we could
measure microstates) would be effectively a random sequence, with each allowed microstate being
equally probable. Restating this in the language of Section 6.1 gives theStatistical Postulate:
When an isolated system is left alone long enough, it evolves to thermal equilib-
rium. Equilibrium is not one microstate, but rather that probability distribu-
tion of microstates having the greatest possible disorder allowed by the physical
constraints on the system.