Pattern Recognition and Machine Learning

10.4. Exponential Family Distributions 491

distribution that factorizes between the latent variables and the parameters, so that

q(Z,η)=q(Z)q(η). Using the general result (10.9), we can solve for the two

factors as follows

lnq
(Z)=Eη[lnp(X,Z|η)]+const

=

∑N

n=1

{

lnh(xn,zn)+E[ηT]u(xn,zn)

}

+const. (10.115)

Thus we see that this decomposes into a sum of independent terms, one for each

value of∏ n, and hence the solution forq
(Z)will factorize overnso thatq
(Z)=

nq

(z
Section 10.2.5 n). This is an example of an induced factorization. Taking the exponential
of both sides, we have

q
(zn)=h(xn,zn)g(E[η]) exp

{

E[ηT]u(xn,zn)

}

(10.116)

where the normalization coefficient has been re-instated by comparison with the

standard form for the exponential family.

Similarly, for the variational distribution over the parameters, we have

lnq
(η)=lnp(η|ν 0 ,χ 0 )+EZ[lnp(X,Z|η)]+const (10.117)

= ν 0 lng(η)+ηTχ 0 +

∑N

n=1

{

lng(η)+ηTEzn[u(xn,zn)]

}

+const. (10.118)

Again, taking the exponential of both sides, and re-instating the normalization coef-

ficient by inspection, we have

q
(η)=f(νN,χN)g(η)νNexp

{

ηTχN

}

(10.119)

where we have defined

νN = ν 0 +N (10.120)

χN = χ 0 +

∑N

n=1

Ezn[u(xn,zn)]. (10.121)

Note that the solutions forq
(zn)andq
(η)are coupled, and so we solve them iter-

atively in a two-stage procedure. In the variational E step, we evaluate the expected

sufficient statisticsE[u(xn,zn)]using the current posterior distributionq(zn)over

the latent variables and use this to compute a revised posterior distributionq(η)over

the parameters. Then in the subsequent variational M step, we use this revised pa-

rameter posterior distribution to find the expected natural parametersE[ηT], which

gives rise to a revised variational distribution over the latent variables.

10.4.1 Variational message passing

We have illustrated the application of variational methods by considering a spe-

cific model, the Bayesian mixture of Gaussians, in some detail. This model can be