Pattern Recognition and Machine Learning

Exercises 601

12.14 (*) Thenumberofindependentparametersinthecovariancematrixfortheproba-
bilisticPCAmodelwithanM-dimensionallatentspaceanda D-dimensionaldata

spaceisgivenby(12.51). VerifythatinthecaseofM = D- 1,thenumberof

independentparametersis thesameasina generalcovarianceGaussian,whereasfor

M =°it is thesameasfora Gaussianwithanisotropiccovariance.

12.15 (**)IIiI!I DerivetheM-stepequations(12.56)and(12.57)fortheprobabilistic
PCAmodelbymaximizationoftheexpectedcomplete-dataloglikelihoodfunction
givenby(12.53).

12.16 (***) InFigure12.11,weshowedanapplicationofprobabilisticPCAtoa dataset
inwhichsomeofthedatavalues weremissingat random.DerivetheEMalgorithm
formaximizingthelikelihoodfunctionfortheprobabilisticPCAmodelinthissitu-
ation.Notethatthe{zn},aswellasthemissingdatavaluesthatarecomponentsof
thevectors{xn},arenowlatentvariables.Showthatinthespecialcaseinwhichall
ofthedatavaluesareobserved,thisreducestotheEMalgorithmforprobabilistic
PCAderivedinSection12.2.2.

12.17 (**)IIiI!I LetW beaD xM matrixwhosecolumnsdefinea linearsubspace
ofdimensionalityM embeddedwithina dataspaceofdimensionalityD,andletJ1

bea D-dimensionalvector. Givena dataset{xn}wheren= 1,...,N,wecan

approximatethedatapointsusinga linearmappingfroma setofM-dimensional

vectors{zn},sothatXnisapproximatedbyWZn+J1. Theassociatedsum-of-

squaresreconstructioncostis givenby

N

J= L Ilxn- J1-Wzn 11

2

.

n=l

(12.95)

FirstshowthatminimizingJwithrespecttoJ1leadstoananalogousexpressionwith

XnandZnreplacedbyzero-meanvariablesXn-xandZn-Z,respectively,wherex

andZ denotesamplemeans.ThenshowthatminimizingJwithrespecttoZn,where

W iskeptfixed,givesrisetothePCAEstep(12.58),andthatminimizingJwith

respecttoW,where{zn}is keptfixed,givesrisetothePCAMstep(12.59).

12.18 (*) Deriveanexpressionforthenumberofindependentparametersinthefactor
analysismodeldescribedinSection12.2.4.

12.19 (**)IIiI!I ShowthatthefactoranalysismodeldescribedinSection12.2.4is
invariantunderrotationsofthelatentspacecoordinates.

12.20 (**) Byconsideringsecondderivatives,showthattheonlystationarypointof
theloglikelihoodfunctionforthefactoranalysismodeldiscussedinSection12.2.4
withrespecttotheparameterJ1isgivenbythesamplemeandefinedby(12.1).
Furthermore,showthatthisstationarypointis a maximum.

12.21 (**) Derivetheformulae(12.66)and(12.67)fortheE stepoftheEMalgorithm
forfactoranalysis.NotethatfromtheresultofExercise 12.20,theparameterJ1can
bereplacedbythesamplemeanx.