Pattern Recognition and Machine Learning

562 12.CONTINUOUSLATENTVARIABLES

chapter,weshallconsidertechniquestodetermineanappropriatevalueofIV!from

thedata.

Tobeginwith,considertheprojectionontoa one-dimensionalspace(M= 1).

Wecandefinethedirectionofthisspaceusinga D-dimensionalvectorUl,which

forconvenience(andwithoutlossofgenerality)weshallchoosetobea unitvector

sothatufUl = 1 (notethatweareonlyinterestedinthedirectiondefinedbyUl,

notinthemagnitudeofUlitself).EachdatapointXnisthenprojectedontoa scalar

valueufXn.Themeanoftheprojecteddataisufxwherex is thesamplesetmean

givenby

(12.1)

andthevarianceoftheprojecteddatais givenby

whereS is thedatacovariancematrixdefinedby

1 N

S= -NLJ"(xn- x)(xn- x)T.

n=l

(12.2)

(12.3)

AppendixE

WenowmaximizetheprojectedvarianceUfSUlwithrespecttoUl.Clearly,thishas

tobea constrainedmaximizationtopreventIlulll.....00.Theappropriateconstraint

comesfromthenormalizationconditionufUl = 1. Toenforcethisconstraint,

weintroducea LagrangemultiplierthatweshalldenotebyAI,andthenmakean

unconstrainedmaximizationof

(12.4)

BysettingthederivativewithrespecttoUlequaltozero,weseethatthisquantity

willhavea stationarypointwhen

( 12.5)

whichsaysthatUlmustbeaneigenvectorofS.Ifweleft-multiplybyufandmake

useofufUl= 1,weseethatthevarianceis givenby

(12.6)

andsothevariancewillbea maximumwhenwesetUlequaltotheeigenvector

havingthelargesteigenvalueAI. Thiseigenvectorisknownasthefirstprincipal

component.

Wecandefineadditionalprincipalcomponentsinanincrementalfashionby

choosingeachnewdirectiontobethatwhichmaximizestheprojectedvariance