Pattern Recognition and Machine Learning

Exercises 599

bilisticfoundationalsomakesit verystraightforwardtodefinegeneralizationsof

GTM(Bishopet al.,1998a)suchasa Bayesiantreatment,dealingwithmissingval-

Section6.4 ues, a principledextensiontodiscretevariables,theuseofGaussianprocessesto

definethemanifold,ora hierarchicalGTMmodel(TinoandNabney,2002).

BecausethemanifoldinGTMis definedasa continuoussurface,notjustatthe

prototypevectorsasintheSOM,itispossibletocomputethemagnificationfactors

correspondingtothelocalexpansionsandcompressionsofthemanifoldneededto

fitthedataset(Bishopetal., 1997b)aswellasthedirectionalcurvaturesofthe

manifold(Tinoetal.,2001).Thesecanbevisualizedalongwiththeprojecteddata

andprovideadditionalinsightintothemodel.

Exercises

12.1 (**)lIB Inthisexercise,weuseproofbyinductiontoshowthatthelinear

projectionontoanM-dimensionalsubspacethatmaximizesthevarianceofthepro-

jecteddataisdefinedbytheM eigenvectorsofthedatacovariancematrixS,given

by(12.3),correspondingtotheM largesteigenvalues. InSection12.1,thisresult

wasprovenforthecaseofM = 1. Nowsupposetheresultholdsforsomegeneral

valueofM andshowthatit consequentlyholdsfordimensionalityM +1. Todo

this,firstsetthederivativeofthevarianceoftheprojecteddatawithrespecttoa
vectorUM+1definingthenewdirectionindataspaceequaltozero. Thisshould
bedonesubjecttotheconstraintsthatUM+lbeorthogonaltotheexistingvectors
U1,"" UM,andalsothat itbenormalizedtounitlength. UseLagrangemultipli-
AppendixE erstoenforcetheseconstraints.Thenmakeuseoftheorthonormalitypropertiesof
thevectorsU1,"" UMtoshowthatthenewvectorUM+1isaneigenvectorofS.
Finally,showthatthevarianceismaximizedif theeigenvectorischosentobethe
onecorrespondingtoeigenvectorAM+1wheretheeigenvalueshavebeenorderedin
decreasingvalue.

12.2 (**) ShowthattheminimumvalueofthePCAdistortionmeasureJ givenby

(12.15)withrespecttotheUi,subjecttotheorthonormalityconstraints(12.7),is

obtainedwhentheUiareeigenvectorsofthedatacovariancematrixS. Todothis,

introducea matrixH ofLagrangemultipliers,oneforeachconstraint,sothatthe

modifieddistortionmeasure,inmatrixnotationreads

] = Tr{UTSU}+Tr{H(I- UTU)} (12.93)

whereUis am~trixofdimensio~Dx(D- M)whosecolumnsaregi:::..enb~Ui.

NowminimizeJwithrespecttoUandshowthatthes~utionsatisfiesSU= UH.

Clearly,onepossiblesolutionisthatthecolumnsofU areeigenvectorsofS,in

whichcaseH isa diagonalmatrixcontainingthecorrespondingeigenvalues. To

obtainthegeneralsolution,showthatHcanbeassumedtobeasymmetr~ma~ix,

andbyusingitseigenvect£rexpansionshowthatthegeneralsolutiontoSU=~UH

givesthesamevalueforJasthespecificsolutioninwhichthecolumnsofU are