Pattern Recognition and Machine Learning

12.1.PrincipalC01n[>OM"1AnalJsis 561

Flgu,e12.2 P'if>cipalcompooonta",,~"seeks"$pace

01 !owe,dimensionality.kt"(>WIlas!heP

pal subSpace"nddenoted I:Jythemagenta "1

line. SUChItlet the Grthogonet[jiojectiOh 01

!hedatapoints('eddoIslontotP'Ns~

"""'imizesthevaria,..,.,of!heproja<:tedpoints

(greendoIs). An"It",nati""....finilion 01 PCA

isbasedonm..mizing the""",-<>I·squares

of!heprojectionerrors.ind'catedbythebfi.>e

lines.

S'crio"12.2 apanlcula,fonnoflinear-Gau"ianlatem"ariablemodel.Thisprobabilisticrefor-

mulationbring~manyad\'imlag~s,su~hastl>l:useI)fEMforparametereslimalion,

rrinciple<Jc~tensioos 10 Oli~turc,ofPeAmodel" andBa)'~sianformulat;onsthat

allowtbenumberofrrincipalcom[>OncntstobedetenninedaUlOmaticallyfrom!be

data.Finally'."cdiSl;us<briefly""'eralgencrali,ation,ofthelatentYariableconcept

thatg<lbe~ood tbelinear-Gaussianassumptionincludingnon·Gau"i"nI.tcntyari-

abies.....hichlea'"totbefr.me....orkofindrl"'mJ.mcompon.nlanal,-.;.,as....ella,

S'di""12.4 modelsha"inga nonlinearrclationshipbet....eenlatentandoose",e<J,'lUiable,.

____'c2=.~1. PrincipalComponentAnalysis

Principalcompooemanaly,;" orrcA.;sa techniquetha!is"'idelyu<edforappli.

cationssuchasdimensionality.-eduction,lossydatacomprc"ion,featuree>tracti"".

anddatav;,ualizatiOll(Jolliffe,2(02).It;salsokno...." astileKaroan.n·I..,;""tran,·

f~.

lbcrcan: t....o commonlyuseddefinitionsofPeAthatgiyerisetothe>arne

algorithm.PeAcanbedefinedastheunhog<lnalprojtttionofthedataO/1tOa lo....er

dimensionallincarspace.kno....nasthepri/lcip.al$uh.•p.aa.sochthatthe\'arianceof

theprojttteddatai'ma~imi,e<J(1I",.lIing.1933).Equi"alemly,;tcanbedefinedas

tbelinearprojectionthatminimi"'.theaverageprojttlioncost.definedast~mean

squa.-eddistance!letweenthedata[>Oint<andtbeirp<ojtttioo,(Pearson,19(1).The

l"J'"OC"s<ofonhogonalprojectioni'illustraledinFiguTe12.2. Wecon,idereachof

thesedefinitionsintum.

12,1.1 Mllximllmvariancelormulation

Con,idera dalaset<Ifobser"\lations{x,,}where" = 1 .....S,andx"i,a

Euclideanvariable"'ilhdimen,ionalityD. OurgoalistoprojectIf>/::dataontoa

'paceha"ingdimen,ionalityM <D"hileIll3Jli",i,illgthe"ariallCeoftheprojttted

data. Forthe!noll..nl.we'hallassumethattbe"alueofMisg;\·en. Latcrinthis