Pattern Recognition and Machine Learning

(Jeff_L) #1
Exercises 599

bilisticfoundationalsomakesit verystraightforwardtodefinegeneralizationsof
GTM(Bishopet al.,1998a)suchasa Bayesiantreatment,dealingwithmissingval-
Section6.4 ues, a principledextensiontodiscretevariables,theuseofGaussianprocessesto
definethemanifold,ora hierarchicalGTMmodel(TinoandNabney,2002).
BecausethemanifoldinGTMis definedasa continuoussurface,notjustatthe
prototypevectorsasintheSOM,itispossibletocomputethemagnificationfactors
correspondingtothelocalexpansionsandcompressionsofthemanifoldneededto
fitthedataset(Bishopetal., 1997b)aswellasthedirectionalcurvaturesofthe
manifold(Tinoetal.,2001).Thesecanbevisualizedalongwiththeprojecteddata
andprovideadditionalinsightintothemodel.

Exercises


12.1 (**)lIB Inthisexercise,weuseproofbyinductiontoshowthatthelinear
projectionontoanM-dimensionalsubspacethatmaximizesthevarianceofthepro-
jecteddataisdefinedbytheM eigenvectorsofthedatacovariancematrixS,given
by(12.3),correspondingtotheM largesteigenvalues. InSection12.1,thisresult
wasprovenforthecaseofM = 1. Nowsupposetheresultholdsforsomegeneral

valueofM andshowthatit consequentlyholdsfordimensionalityM +1. Todo


this,firstsetthederivativeofthevarianceoftheprojecteddatawithrespecttoa
vectorUM+1definingthenewdirectionindataspaceequaltozero. Thisshould
bedonesubjecttotheconstraintsthatUM+lbeorthogonaltotheexistingvectors
U1,"" UM,andalsothat itbenormalizedtounitlength. UseLagrangemultipli-
AppendixE erstoenforcetheseconstraints.Thenmakeuseoftheorthonormalitypropertiesof
thevectorsU1,"" UMtoshowthatthenewvectorUM+1isaneigenvectorofS.
Finally,showthatthevarianceismaximizedif theeigenvectorischosentobethe
onecorrespondingtoeigenvectorAM+1wheretheeigenvalueshavebeenorderedin
decreasingvalue.


12.2 (**) ShowthattheminimumvalueofthePCAdistortionmeasureJ givenby
(12.15)withrespecttotheUi,subjecttotheorthonormalityconstraints(12.7),is
obtainedwhentheUiareeigenvectorsofthedatacovariancematrixS. Todothis,
introducea matrixH ofLagrangemultipliers,oneforeachconstraint,sothatthe
modifieddistortionmeasure,inmatrixnotationreads

] = Tr{UTSU}+Tr{H(I- UTU)} (12.93)


whereUis am~trixofdimensio~Dx(D- M)whosecolumnsaregi:::..enb~Ui.


NowminimizeJwithrespecttoUandshowthatthes~utionsatisfiesSU= UH.


Clearly,onepossiblesolutionisthatthecolumnsofU areeigenvectorsofS,in
whichcaseH isa diagonalmatrixcontainingthecorrespondingeigenvalues. To
obtainthegeneralsolution,showthatHcanbeassumedtobeasymmetr~ma~ix,

andbyusingitseigenvect£rexpansionshowthatthegeneralsolutiontoSU=~UH


givesthesamevalueforJasthespecificsolutioninwhichthecolumnsofU are
Free download pdf