commonly used functions in modeling hidden units is the logistic function
(figure 19.9).
This nonlinearity of hidden units enables a network to implement mappings
from input to expectation that would otherwise be impossible. (There is no ad-
vantage to hidden units if they are linear). If the tips of the tonal composite
vectors that generate an expectation for pitch classxcannot be clearly sepa-
rated by a hyperplane from the tips of the tonal composite vectors that generate
an expectation for pitch classy, then the expectations are notlinearly separable.
In other words, if there are cases in which similar tonal composites generate
different expectations and dissimilar tonal composites generate the same ex-
pectation, then the expectations may not be linearly separable. Similar tonal
composites tend to generate similar schematic expectancies but not necessarily
similar veridical expectancies. This is because composers occasionally use un-
usual or unschematic transitions that violate (schematic) expectations for aes-
thetic effect (Meyer, 1956). Problems that are not linearly separable cannot be
solved by neural nets without nonlinear hidden units (Minsky & Papert, 1969)
or extra assumptions.
We use the logistic function at the expectation units as well because it has
theeffectofmakingtheactivationsattheexpectationunitsequivalenttoprob-
abilities. The weights in the network are initially random. As a sequence is
played, a temporal composite at the input produces a pattern of expectations
that is initially random. The network learns by comparing the expectation for
the next event with the actual next event when it occurs. Each event thus trains
the expectations that attempted to predict it.
The delta rule is adapted for this model as follows. The error signal is scaled
by the slope of the logistic function at the expectation unit’s current activation
level (the derivative of the activation ofewith respect to its net input):
de¼ðteaeÞdae
dnete;
Figure 19.8
A network that learns individual sequences (veridical expectancies) and acquires schematic prop-
erties. The ‘‘Additional Context’’ units represent any extratonal information that may be encoded as
context. The arrows between groups of units represent a link from each unit of one group to each
unit of the other.
472 Jamshed J. Bharucha