Pattern Recognition and Machine Learning

10.2. Illustration: Variational Mixture of Gaussians 481 Indeed, these singularities are removed if we simply introduce a prior ...

482 10. APPROXIMATE INFERENCE E[lnp(μ,Λ)] = 1 2 ∑K k=1 { Dln(β 0 / 2 π)+ln ̃Λk− Dβ 0 βk −β 0 νk(mk−m 0 )TWk(mk−m 0 ) } +KlnB(W 0 ...

10.2. Illustration: Variational Mixture of Gaussians 483 wherep(π,μ,Λ|X)is the (unknown) true posterior distribution of the para ...

484 10. APPROXIMATE INFERENCE Figure 10.7 Plot of the variational lower bound Lversus the numberKof com- ponents in the Gaussian ...

10.2. Illustration: Variational Mixture of Gaussians 485 to explaining the data will have their mixing coefficients driven to ze ...

486 10. APPROXIMATE INFERENCE is satisfied. We can test to see if this relation does hold, for any choice ofAandB by making use ...

10.3. Variational Linear Regression 487 Figure 10.8 Probabilistic graphical model representing the joint dis- tribution (10.90) ...

488 10. APPROXIMATE INFERENCE where mN = βSNΦTt (10.100) SN = ( E[α]I+βΦTΦ )− 1 . (10.101) Note the close similarity to the post ...

10.3. Variational Linear Regression 489 where we have evaluated the integral by making use of the result (2.115) for the linear- ...

490 10. APPROXIMATE INFERENCE Figure 10.9 Plot of the lower boundL ver- sus the orderM of the polyno- mial, for a polynomial mod ...

10.4. Exponential Family Distributions 491 distribution that factorizes between the latent variables and the parameters, so that ...

492 10. APPROXIMATE INFERENCE described by the directed graph shown in Figure 10.5. Here we consider more gen- erally the use of ...

10.5. Local Variational Methods 493 10.5 Local Variational Methods The variational framework discussed in Sections 10.1 and 10.2 ...

494 10. APPROXIMATE INFERENCE x y f(x) λx x y f(x) λx−g(λ) −g(λ) Figure 10.11 In the left-hand plot the red curve shows a convex ...

10.5. Local Variational Methods 495 Now, instead of fixingλand varyingx, we can consider a particularxand then adjustλuntil the ...

496 10. APPROXIMATE INFERENCE λ=0. 2 λ=0. 7 − 6 0 6 0 0.5 1 ξ=2. 5 −6 −ξξ 0 6 0 0.5 1 Figure 10.12 The left-hand plot shows the ...

10.5. Local Variational Methods 497 Instead of thinking ofλas the variational parameter, we can letξplay this role as this leads ...

498 10. APPROXIMATE INFERENCE Although the boundσ(a)f(a, ξ)on the logistic sigmoid can be optimized exactly, the required choic ...

10.6. Variational Logistic Regression 499 we reproduce here for convenience σ(z)σ(ξ)exp { (z−ξ)/ 2 −λ(ξ)(z^2 −ξ^2 ) } (10.149) ...

500 10. APPROXIMATE INFERENCE This is a quadratic function ofw, and so we can obtain the corresponding variational approximation ...