Pattern Recognition and Machine Learning

12.1.PrincipalComponentAnalysis 565

minimizeJ=UISU2'subjecttothenormalizationconstraintuIU2=1.Usinga

LagrangemultiplierA2toenforcetheconstraint,weconsidertheminimizationof

(12.16)

(12.18)

Exercise 12.2

AppendixA

SettingthederivativewithrespecttoU2tozero,weobtainSU2= A2U2sothatU2

isaneigenvectorofS witheigenvalueA2. Thusanyeigenvectorwilldefinea sta-

tionarypointofthedistortionmeasure.TofindthevalueofJattheminimum,we

back-substitutethesolutionforU2intothedistortionmeasuretogiveJ= A2.We

thereforeobtaintheminimumvalueofJbychoosingU2tobetheeigenvectorcorre-

spondingtothesmallerofthetwoeigenvalues.Thusweshouldchoosetheprincipal

subspacetobealignedwiththeeigenvectorhavingthelargereigenvalue.Thisresult

accordswithourintuitionthat,inordertominimizetheaveragesquaredprojection

distance,weshouldchoosetheprincipalcomponentsubspacetopassthroughthe

meanofthedatapointsandtobealignedwiththedirectionsofmaximumvariance.

Forthecasewhentheeigenvaluesareequal,anychoiceofprincipal directionwill

giverisetothesamevalueofJ.

ThegeneralsolutiontotheminimizationofJforarbitraryDandarbitraryM <

Dis obtainedbychoosingthe{Ui}tobeeigenvectorsofthecovariancematrixgiven

by

SUi= AiUi (12.17)

wherei =1,...,D,andasusualtheeigenvectors{Ui}arechosentobeorthonor-

mal.Thecorrespondingvalueofthedistortionmeasureis thengivenby

D

J= L Ai

i=M+l

whichis simplythesumoftheeigenvaluesofthoseeigenvectorsthatareorthogonal

totheprincipalsubspace.WethereforeobtaintheminimumvalueofJbyselecting

theseeigenvectorstobethosehavingtheD- M smallesteigenvalues,andhence

theeigenvectorsdefiningtheprincipalsubspacearethosecorrespondingtotheM

largesteigenvalues.

AlthoughwehaveconsideredM < D,thePCAanalysisstillholdsifM =

D,inwhichcasethereisnodimensionalityreductionbutsimplya rotationofthe

coordinateaxestoalignwithprincipalcomponents.

Finally,it is worthnotingthatthereexistsa closelyrelatedlineardimensionality

reductiontechniquecalledcanonicalcorrelationanalysis,orCCA(Hotelling,1936;

BachandJordan,2002).WhereasPCAworkswitha singlerandomvariable,CCA

considerstwo(ormore)variablesandtriestofinda correspondingpairoflinear

subspacesthathavehigh cross-correlation,sothateachcomponentwithinoneofthe

subspacesis correlatedwitha singlecomponentfromtheothersubspace.Itssolution

canbeexpressedintermsofa generalizedeigenvectorproblem.

12.1.3 ApplicationsofpeA

WecanillustratetheuseofPCAfordatacompressionbyconsideringtheoff-

linedigitsdataset. Becauseeacheigenvectorofthecovariancematrixisa vector