Pattern Recognition and Machine Learning

10 Approximate Inference A central task in the application of probabilistic models is the evaluation of the pos- terior distribu ...

462 10. APPROXIMATE INFERENCE analytical solutions, while the dimensionality of the space and the complexity of the integrand ma ...

10.1. Variational Inference 463 as the output. We can the introduce the concept of afunctional derivative, which ex- presses how ...

464 10. APPROXIMATE INFERENCE −2 −1 0 1 2 3 4 0 0.2 0.4 0.6 0.8 1 −2 −1 0 1 2 3 4 0 10 20 30 40 Figure 10.1 Illustration of the ...

10.1. Variational Inference 465 It should be emphasized that we are making no further assumptions about the distri- bution. In p ...

466 10. APPROXIMATE INFERENCE divergence, and the minimum occurs whenqj(Zj)= ̃p(X,Zj). Thus we obtain a general expression for t ...

10.1. Variational Inference 467 optimal factorq 1
(z 1 ). In doing so it is useful to note that on the right-hand side we only ...

468 10. APPROXIMATE INFERENCE Figure 10.2 Comparison of the two alternative forms for the Kullback-Leibler divergence. The green ...

10.1. Variational Inference 469 (a) (b) (c) Figure 10.3 Another comparison of the two alternative forms for the Kullback-Leibler ...

470 10. APPROXIMATE INFERENCE of divergences (Ali and Silvey, 1966; Amari, 1985; Minka, 2005) defined by Dα(p‖q)= 4 1 −α^2 ( 1 − ...

10.1. Variational Inference 471 Note that the true posterior distribution does not factorize in this way. The optimum factorsqμ( ...

472 10. APPROXIMATE INFERENCE μ τ (a) −1 0 1 0 1 2 μ τ (b) −1 0 1 0 1 2 μ τ (c) −1 0 1 0 1 2 μ τ (d) −1 0 1 0 1 2 Figure 10.4 Il ...

10.1. Variational Inference 473 qμ(μ)in the form E[μ]=x, E[μ^2 ]=x^2 + 1 NE[τ] . (10.32) Exercise 10.9 We can now substitute the ...

474 10. APPROXIMATE INFERENCE of (10.35), and then subsequently determining theq(m)using (10.36). After nor- malization the resu ...

10.2. Illustration: Variational Mixture of Gaussians 475 Figure 10.5 Directed acyclic graph representing the Bayesian mix- ture ...

476 10. APPROXIMATE INFERENCE We now consider a variational distribution which factorizes between the latent variables and the p ...

10.2. Illustration: Variational Mixture of Gaussians 477 where rnk= ρnk ∑K j=1 ρnj . (10.49) We see that the optimal solution fo ...

478 10. APPROXIMATE INFERENCE Identifying the terms on the right-hand side of (10.54) that depend onπ,wehave lnq
(π)=(α 0 −1) ∑K ...

10.2. Illustration: Variational Mixture of Gaussians 479 where we have introduced definitions ofΛ ̃kand ̃πk, andψ(·)is the digam ...

480 10. APPROXIMATE INFERENCE Figure 10.6 Variational Bayesian mixture ofK =6Gaussians ap- plied to the Old Faithful data set, i ...