Pattern Recognition and Machine Learning

10.6. Variational Logistic Regression 501 we then use these parameter values to find the posterior distribution overw, which is ...

502 10. APPROXIMATE INFERENCE 0.010.250.75 0.99 −4 −2 0 2 4 −6 −4 −2 0 2 4 6 −4 −2 0 2 4 −6 −4 −2 0 2 4 6 Figure 10.13 Illustrat ...

10.6. Variational Logistic Regression 503 Specifically, we consider once again a simple isotropic Gaussian prior distribu- tion ...

504 10. APPROXIMATE INFERENCE With this factorization we can appeal to the general result (10.9) to find expressions for the opt ...

10.7. Expectation Propagation 505 We also need to optimize the variational parametersξn, and this is also done by maximizing the ...

506 10. APPROXIMATE INFERENCE We see that the optimum solution simply corresponds to matching the expected suf- ficient statisti ...

10.7. Expectation Propagation 507 be described by a finite set of sufficient statistics. For example, if each of the ̃fi(θ) is a ...

508 10. APPROXIMATE INFERENCE −2 −1 0 1 2 3 4 0 0.2 0.4 0.6 0.8 1 −2 −1 0 1 2 3 4 0 10 20 30 40 Figure 10.14 Illustration of the ...

10.7. Expectation Propagation 509 sides of (10.199) byq\i(θ)and integrating to give K= ∫ ̃fj(θ)q\j(θ)dθ (10.200) where we have u ...

510 10. APPROXIMATE INFERENCE (c) Evaluate the new posterior by setting the sufficient statistics (moments) ofqnew(θ)equal to th ...

10.7. Expectation Propagation 511 Figure 10.15 Illustration of the clutter problem for a data space dimensionality of D=1. Train ...

512 10. APPROXIMATE INFERENCE The factor approximations will therefore take the form of exponential-quadratic functions of the f ...

10.7. Expectation Propagation 513 −5 (^05) θ 10 −5 (^05) θ 10 Figure 10.16 Examples of the approximation of specific factors for ...

514 10. APPROXIMATE INFERENCE ep laplace vb Posterior mean FLOPS Error 104 106 10 −5 100 ep vb laplace Evidence FLOPS Error 104 ...

10.7. Expectation Propagation 515 x 1 x 2 x 3 x 4 fa fb fc x 1 x 2 x 3 x 4 f ̃a 1 f ̃a 2 f ̃b 2 f ̃b 3 f ̃c 2 f ̃c 4 Figure 10.1 ...

516 10. APPROXIMATE INFERENCE Section 8.4.4 These are precisely the messages obtained using belief propagation in which mes- sag ...

Exercises 517 We recognize this as the sum-product rule in the form in which messages from vari- able nodes to factor nodes have ...

518 10. APPROXIMATE INFERENCE minimization ofKL(p‖q)with respect toμandΣleads to the result thatμis given by the expectation ofx ...

Exercises 519 10.13 ( ) www Starting from (10.54), derive the result (10.59) for the optimum vari- ational posterior distributio ...

520 10. APPROXIMATE INFERENCE 10.22 ( ) We have seen that each mode of the posterior distribution in a Gaussian mix- ture model ...