Pattern Recognition and Machine Learning

Exercises 321

6.11 ( ) By making use of the expansion (6.25), and then expanding the middle factor
as a power series, show that the Gaussian kernel (6.23) can be expressed as the inner
product of an infinite-dimensional feature vector.

6.12 ( ) www Consider the space of all possible subsetsAof a given fixed setD.
Show that the kernel function (6.27) corresponds to an inner product in a feature
space of dimensionality 2 |D|defined by the mappingφ(A)whereAis a subset ofD
and the elementφU(A), indexed by the subsetU, is given by

φU(A)=

{

1 , ifU⊆A;

0 , otherwise.

(6.95)

HereU⊆Adenotes thatUis either a subset ofAor is equal toA.

6.13 ( ) Show that the Fisher kernel, defined by (6.33), remains invariant if we make
a nonlinear transformation of the parameter vectorθ→ψ(θ), where the function
ψ(·)is invertible and differentiable.

6.14 ( ) www Write down the form of the Fisher kernel, defined by (6.33), for the
case of a distributionp(x|μ)=N(x|μ,S)that is Gaussian with meanμand fixed
covarianceS.

6.15 ( ) By considering the determinant of a 2 × 2 Gram matrix, show that a positive-
definite kernel functionk(x, x′)satisfies the Cauchy-Schwartz inequality

k(x 1 ,x 2 )^2 
k(x 1 ,x 1 )k(x 2 ,x 2 ). (6.96)

6.16 ( ) Consider a parametric model governed by the parameter vectorwtogether
with a data set of input valuesx 1 ,...,xNand a nonlinear feature mappingφ(x).
Suppose that the dependence of the error function onwtakes the form

J(w)=f(wTφ(x 1 ),...,wTφ(xN)) +g(wTw) (6.97)

whereg(·)is a monotonically increasing function. By writingwin the form

w=

∑N

n=1

αnφ(xn)+w⊥ (6.98)

show that the value ofwthat minimizesJ(w)takes the form of a linear combination

of the basis functionsφ(xn)forn=1,...,N.

6.17 ( ) www Consider the sum-of-squares error function (6.39) for data having
noisy inputs, whereν(ξ)is the distribution of the noise. Use the calculus of vari-
ations to minimize this error function with respect to the functiony(x), and hence
show that the optimal solution is given by an expansion of the form (6.40) in which
the basis functions are given by (6.41).