Pattern Recognition and Machine Learning

586 12.CONTINUOUSLATENTVARIABLES

Exercise 12.22

tocomputeinO(D)steps),whichis convenientbecauseoftenM « D.Similarly,

theM-step equationstaketheform

wnew [~(x"-XllllIZn]"] [~Ill[Znz~I]-'

diag{s-W.'w~~1ll[Zn](Xn_xl"}

(12.69)

(12.70)

wherethe'diag'operatorsetsallofthenondiagonalelementsofa matrixtozero.A
Bayesiantreatmentofthefactoranalysismodelcanbeobtainedbya straightforward
applicationofthetechniquesdiscussedinthisbook.
AnotherdifferencebetweenprobabilisticPCAandfactoranalysisconcernstheir
Exercise 12.25 differentbehaviourundertransformationsofthedataset.ForPCAandprobabilis-
ticPCA,if werotatethecoordinatesystemindataspace,thenweobtainexactly
thesamefittothedatabutwiththeW matrixtransformedbythecorresponding
rotationmatrix. However,forfactoranalysis,theanalogouspropertyisthatifwe
makea component-wisere-scalingofthedatavectors,thenthisisabsorbedintoa
correspondingre-scalingoftheelementsof)i.

12.3. KernelpeA

InChapter6,wesawhowthetechniqueofkernelsubstitutionallowsustotakean

algorithm expressedintermsofscalarproductsoftheformxTx'andgeneralize

thatalgorithmbyreplacingthescalarproductswitha nonlinearkernel. Herewe

applythistechniqueofkernelsubstitutiontoprincipalcomponentanalysis,thereby

obtaininga nonlineargeneralizationcalledkernelpeA(Scholkopfetal.,1998).

Considera dataset{xn}ofobservations,wheren = 1,...,N,ina spaceof

dimensionalityD. Inordertokeepthenotationuncluttered,weshallassumethat

wehavealreadysubtractedthesamplemeanfromeachofthevectorsXn,sothat

LnXn= O. ThefirststepistoexpressconventionalPCAinsucha formthatthe

datavectors{xn}appearonlyintheformofthescalar productsx~Xm.Recallthat

theprincipalcomponentsaredefinedbytheeigenvectorsUiofthecovariancematrix

SUi= AiUi (12.71)

wherei= 1,...,D.HeretheDxDsample covariancematrixSisdefinedby

(12.72)

andtheeigenvectorsarenormalizedsuchthatuTUi= 1.

Nowconsidera nonlineartransformation¢(x)intoanM-dimensionalfeature

space,sothateachdatapointXnistherebyprojectedontoa point¢(xn).Wecan