Pattern Recognition and Machine Learning

5.5. Regularization in Neural Networks 263

Figure 5.14 Illustration of the synthetic warping of a handwritten digit. The original image is shown on the
left. On the right, the top row shows three examples of warped digits, with the corresponding displacement
fields shown on the bottom row. These displacement fields are generated by sampling random displacements
∆x,∆y∈(0,1)at each pixel and then smoothing by convolution with Gaussians of width 0. 01 , 30 and 60
respectively.

One advantage of approach 3 is that it can correctly extrapolate well beyond the

range of transformations included in the training set. However, it can be difficult

to find hand-crafted features with the required invariances that do not also discard

information that can be useful for discrimination.

5.5.4 Tangent propagation

We can use regularization to encourage models to be invariant to transformations

of the input through the technique oftangent propagation(Simardet al., 1992).

Consider the effect of a transformation on a particular input vectorxn. Provided the

transformation is continuous (such as translation or rotation, but not mirror reflection

for instance), then the transformed pattern will sweep out a manifoldMwithin the

D-dimensional input space. This is illustrated in Figure 5.15, for the case ofD=

2 for simplicity. Suppose the transformation is governed by a single parameterξ

(which might be rotation angle for instance). Then the subspaceMswept out byxn

Figure 5.15 Illustration of a two-dimensional input space

showing the effect of a continuous transforma-

tion on a particular input vectorxn. A one-

dimensional transformation, parameterized by

the continuous variableξ, applied toxncauses

it to sweep out a one-dimensional manifoldM.

Locally, the effect of the transformation can be

approximated by the tangent vectorτn.

x 1

x 2

xn

τn

ξ

M