Pattern Recognition and Machine Learning

576 12.CONTINUOUSLATENTVARIABLES

Therotationalinvarianceinlatentspacerepresentsa formofstatisticalnoniden-

tifiability,analogoustothatencounteredformixturemodelsinthecaseofdiscrete

latentvariables. Herethereisa continuumofparametersallofwhichleadtothe

samepredictivedensity,incontrasttothediscretenonidentifiabilityassociatedwith

componentre-labellinginthemixturesetting.

IfweconsiderthecaseofM =D,sothatthereisnoreductionofdimension-

ality,thenUM = UandLM = L. Makinguseoftheorthogonalityproperties

UUT =IandRRT=I,weseethatthecovarianceCofthemarginaldistribution

forx becomes

(12.47)

andsoweobtainthestandardmaximumlikelihoodsolutionfor anunconstrained

Gaussiandistributioninwhichthecovariancematrixis givenbythesamplecovari-

ance.

ConventionalPCAis generallyformulatedasa projectionofpointsfromtheD-

dimensionaldataspaceontoanM-dimensionallinearsubspace.ProbabilisticPCA,

however,is mostnaturallyexpressedasa mappingfromthelatentspaceintothedata

spacevia(12.33).Forapplicationssuchasvisualizationanddatacompression,we

canreversethismappingusingBayes'theorem.Anypointxindataspacecanthen

besummarizedbyitsposteriormeanandcovarianceinlatentspace.From(12.42)

themeanis givenby

whereMis givenby(12.41).Thisprojectstoa pointindataspacegivenby

WlE[zlx]+J-L.

(12.48)

(12.49)

Section3.3.1 Notethatthistakesthesameformastheequationsforregularizedlinearregression

andisa consequenceofmaximizingthelikelihoodfunctionfora linearGaussian

model. Similarly,theposteriorcovarianceisgivenfrom(12.42)by0-2M-^1 andis

independentofx.

Ifwetakethelimit0-^2 ----t0,thentheposteriormeanreducesto

(12.50)

Exercise 12.11

Exercise 12.12

Section2.3

whichrepresentsanorthogonalprojectionofthedatapointontothelatentspace,

andsowerecoverthestandardPCAmodel.Theposteriorcovarianceinthislimitis

zero,however,andthe densitybecomessingular.For0-^2 >0,thelatentprojection

is shiftedtowardstheorigin,relativetotheorthogonalprojection.

Finally,wenotethatanimportantrolefortheprobabilisticPCAmodelisin

defininga multivariateGaussiandistributioninwhichthenumberofdegreesoffree-

dom,inotherwordsthenumberofindependentparameters,canbecontrolledwhilst

stillallowingthemodel tocapturethedominantcorrelationsinthedata. Recall

thata generalGaussiandistributionhasD(D+1)/2independentparametersinits

covariancematrix(plusanotherDparametersinits mean). Thusthenumberof

parametersscalesquadraticallywithDandcanbecomeexcessiveinspacesofhigh