Pattern Recognition and Machine Learning

214 4. LINEAR MODELS FOR CLASSIFICATION

over the parameter vectorwsince the posterior distribution is no longer Gaussian.
It is therefore necessary to introduce some form of approximation. Later in the
Chapter 10 book we shall consider a range of techniques based on analytical approximations

Chapter 11 and numerical sampling.
Here we introduce a simple, but widely used, framework called the Laplace ap-
proximation, that aims to find a Gaussian approximation to a probability density
defined over a set of continuous variables. Consider first the case of a single contin-
uous variablez, and suppose the distributionp(z)is defined by

p(z)=

1

Z

f(z) (4.125)

whereZ =

∫

f(z)dzis the normalization coefficient. We shall suppose that the

value ofZis unknown. In the Laplace method the goal is to find a Gaussian approx-

imationq(z)which is centred on a mode of the distributionp(z). The first step is to

find a mode ofp(z), in other words a pointz 0 such thatp′(z 0 )=0, or equivalently

df(z)

dz

∣

∣

∣

∣

z=z 0

=0. (4.126)

A Gaussian distribution has the property that its logarithm is a quadratic function

of the variables. We therefore consider a Taylor expansion oflnf(z)centred on the

modez 0 so that

lnf(z)lnf(z 0 )−

1

2

A(z−z 0 )^2 (4.127)

where

A=−

d^2

dz^2

lnf(z)

∣

∣

∣

∣

z=z 0

. (4.128)

Note that the first-order term in the Taylor expansion does not appear sincez 0 is a

local maximum of the distribution. Taking the exponential we obtain

f(z)f(z 0 )exp

{

−

A

2

(z−z 0 )^2

}

. (4.129)

We can then obtain a normalized distributionq(z)by making use of the standard

result for the normalization of a Gaussian, so that

q(z)=

(

A

2 π

) 1 / 2

exp

{

−

A

2

(z−z 0 )^2

}

. (4.130)

The Laplace approximation is illustrated in Figure 4.14. Note that the Gaussian

approximation will only be well defined if its precisionA> 0 , in other words the

stationary pointz 0 must be a local maximum, so that the second derivative off(z)

at the pointz 0 is negative.