5.4. The Hessian Matrix 255
usual by summing over the contributions from each of the patterns separately. For
the two-layer network, the forward-propagation equations are given by
aj =
∑
i
wjixi (5.98)
zj = h(aj) (5.99)
yk =
∑
j
wkjzj. (5.100)
We now act on these equations using theR{·}operator to obtain a set of forward
propagation equations in the form
R{aj} =
∑
i
vjixi (5.101)
R{zj} = h′(aj)R{aj} (5.102)
R{yk} =
∑
j
wkjR{zj}+
∑
j
vkjzj (5.103)
wherevjiis the element of the vectorvthat corresponds to the weightwji. Quan-
tities of the formR{zj},R{aj}andR{yk}are to be regarded as new variables
whose values are found using the above equations.
Because we are considering a sum-of-squares error function, we have the fol-
lowing standard backpropagation expressions:
δk = yk−tk (5.104)
δj = h′(aj)
∑
k
wkjδk. (5.105)
Again, we act on these equations with theR{·}operator to obtain a set of backprop-
agation equations in the form
R{δk} = R{yk} (5.106)
R{δj} = h′′(aj)R{aj}
∑
k
wkjδk
+h′(aj)
∑
k
vkjδk+h′(aj)
∑
k
wkjR{δk}. (5.107)
Finally, we have the usual equations for the first derivatives of the error
∂E
∂wkj
= δkzj (5.108)
∂E
∂wji
= δjxi (5.109)