Pattern Recognition and Machine Learning

12.2.ProbabilisticpeA

wheretheDxDcovariancematrixCis definedby

C=WWT+0-^2 1.

573

(12.36)

Thisresultcanalsobederivedmoredirectlybynotingthatthepredictivedistribution

willbeGaussianandthenevaluatingitsmeanandcovarianceusing(12.33). This

gives

IE[x]

cov[x]

IE[Wz+JL+E]= JL

IE[(Wz+E)(WZ+E)T]

IE[WZZTWT]+IE[EET]= WWT+0-^21

(12.37)

(12.38)

wherewehaveusedthefactthatz andEareindependentrandomvariablesandhence

areuncorrelated.

Intuitively,wecanthinkofthedistributionp(x)asbeingdefinedbytakingan

isotropicGaussian'spraycan'andmoving itacrosstheprincipalsubspacespraying

Gaussianinkwithdensitydeterminedby0-^2 andweightedbythepriordistribution.

Theaccumulatedinkdensitygivesrisetoa 'pancake'shapeddistributionrepresent-

ingthemarginaldensityp(x).

Thepredictivedistributionp(x)isgovernedbytheparametersJL,W,and0-^2 •

However,thereis redundancyinthisparameterizationcorrespondingtorotationsof

thelatentspacecoordinates.Toseethis,considera matrixW = WRwhereRis

anorthogonalmatrix. UsingtheorthogonalitypropertyRRT =I,weseethatthe

quantityWWTthatappearsinthecovariancematrixCtakestheform

(12.39)

(12.41)

Exercise 12.8

andhenceisindependentofR.Thusthereisa wholefamilyofmatricesWallof

whichgiverisetothesamepredictivedistribution.Thisinvariancecanbeunderstood

intermsofrotationswithinthelatentspace.Weshallreturntoa discussionofthe

numberofindependentparametersinthismodellater.

Whenweevaluatethepredictivedistribution,werequireC-^1 ,whichinvolves

theinversionofaDxDmatrix.Thecomputationrequiredtodothiscanbereduced

bymakinguseofthematrixinversionidentity(C.7)togive

C-^1 =0--^1 1 - 0--2WM-^1 WT (12.40)

wheretheM x MmatrixMis definedby

M =WTW+0-^2 1.

BecauseweinvertMratherthaninvertingCdirectly,thecostofevaluatingC-^1 is

reducedfromO(D^3 )toO(M^3 ).

Aswellasthepredictivedistributionp(x),wewillalsorequiretheposterior

distributionp(zlx),whichcanagainbewrittendowndirectlyusingtheresult(2.116)

forlinear-Gaussianmodelstogive

(12.42)

Notethattheposteriormeandependsonx,whereas theposteriorcovarianceisin-

dependentofx.