Pattern Recognition and Machine Learning

594 12.CONTINUOUSLATENTVARIABLES

Figure12.19 Addition of extra hidden lay-

ersofnoolinearunitsgivesan

auloassocialivenetworkwhich

canperforma noolineardimen-

siooalityreduction.

inputs

x,

F,

F,

outputs

x,

Thesituationis different,however.ifadditionalhiddenlayersarepcrmillcdin

thenetwork.Considerthefour-layerautoassociativcnetworkshowninFigure12.19.

Againtheoutputunitsarelinear,andtheMunitsinthesecondhiddenlayercanalso

belinear.however,thefirstandthirdhiddenlayershavesigmoidalnonlinearactiva-

tionfunctions. Thenetworkis againtrainedbyminimizationoftheerrorfunction

(12.91). Wecanviewthisnetworkas twosuccessivefunctionalmappingsF]and

F 2 ,asindicatedinFigure12.19. ThefirstmappingF] projectstheoriginalD-

dimensionaldataontoanAI-dimensionalsubspaceSdefinedbytheactivationsof

theunitsinthesecondhiddenlayer.Becauseofthepresenceofthefirsthiddenlayer

ofnonlinearunits.thismappingis verygeneral.andinparticularisnotrestrictedto

beinglinear.Similarly.thesecondhalfofthenetworkdefinesanarbitraryfunctional

mappingfromtheM-dimensionalspacebackintotheoriginalD-dimensionalinput

space.Thishasa simplegeometricalinterpretation.asindicatedforthecaseD= 3

andM = 2 inFigure12.20.

Sucha networkeffectivelyperfonnsa nonlinearprincipalcomponentanalysis.

X3

F, "

x,

"

Figure12.20 Geometricalinterpretationofthemappingsperformedbythenetworkin Figure12.1g forthecase
of 0 = 3 inputsandAI= 2 unitsinthemiddlehiddenlayer. ThefunctionF,mapsfromanM-dimensional
spaceSintoa D-dimensiooalspaceandthereforedefinesthewayin whichthespaceSis embeddedwithinthe
originalx-space.SincethemappingF,canber"I()(llinear,theembedding 01 Scanbenonplanar,asindicated
inthefigure.ThemappingF.thendefinesa projectiorlofpointsintheoriginalD-dimensionalspaceintothe
M-dimensionalsubspaceS.