Pattern Recognition and Machine Learning

12.4.NonlinearLatentVariableModels 591

12.4 Nonlinear Latent Variable Models

Exercise 12.28

Inthischapter,wehavefocussedonthesimplestclassofmodelshavingcontinuous

latentvariables,namelythosebasedonlinear-Gaussiandistributions. Aswellas

havinggreatpracticalimportance,thesemodelsarerelativelyeasytoanalyseand

tofittodataandcanalsobeusedascomponentsinmorecomplexmodels. Here

weconsiderbrieflysomegeneralizationsofthisframeworktomodelsthatareeither

nonlinearornon-Gaussian,orboth.

Infact,theissuesofnonlinearityandnon-Gaussianityarerelatedbecausea

generalprobabilitydensitycanbeobtainedfroma simplefixedreferencedensity,

suchasa Gaussian,bymakinga nonlinearchangeofvariables.Thisideaformsthe

basisofseveralpracticallatentvariablemodelsasweshallseeshortly.

12.4.1 Independent component analysis

Webeginbyconsideringmodelsinwhichtheobservedvariablesarerelated

linearlytothelatentvariables,butforwhichthelatentdistributionis non-Gaussian.

Animportantclassofsuchmodels,knownasindependentcomponentanalysis,or

leA,ariseswhenweconsidera distributionoverthelatentvariablesthatfactorizes,

sothat

M

p(z)= IIp(Zj).

j=l

(12.89)

Tounderstandtheroleofsuchmodels,considera situationinwhichtwopeople

aretalkingatthesametime,andwerecordtheirvoices usingtwomicrophones.

Ifweignoreeffectssuchastimedelayandechoes,thenthesignalsreceivedby

themicrophonesatanypointintimewillbegivenbylinearcombinationsofthe

amplitudesofthetwovoices. Thecoefficientsofthislinearcombinationwillbe

constant,andif wecaninfertheirvaluesfromsampledata,thenwecaninvertthe

mixingprocess(assuming itisnonsingular)andtherebyobtaintwocleansignals

eachofwhichcontainsthevoiceofjustoneperson.Thisis anexampleofa problem

calledblindsourceseparationinwhich'blind'referstothefactthatwearegiven

onlythemixeddata,andneithertheoriginalsourcesnorthemixingcoefficientsare

observed(Cardoso,1998).

Thistypeofproblemis sometimesaddressedusingthefollowingapproach

(MacKay,2003)inwhichweignorethetemporalnatureofthesignalsand treatthe

successivesamplesasi.i.d.Weconsidera generativemodelinwhichtherearetwo

latentvariablescorrespondingtotheunobservedspeechsignalamplitudes,andthere

aretwoobservedvariablesgivenbythesignalvaluesat themicrophones.Thelatent

variableshavea jointdistributionthatfactorizesasabove,andtheobservedvariables

aregivenbya linearcombinationofthelatentvariables.Thereis noneedtoinclude

a noisedistributionbecausethenumberoflatentvariablesequalsthenumberofob-

servedvariables,andthereforethemarginaldistributionoftheobservedvariables

willnotingeneralbesingular,sotheobservedvariablesaresimplydeterministic

linearcombinationsofthelatentvariables. Givena datasetofobservations,the