Pattern Recognition and Machine Learning

3.1. Linear Basis Function Models 141 will be simply E[t|x]= ∫ tp(t|x)dt=y(x,w). (3.9) Note that the Gaussian noise assumption i ...

142 3. LINEAR MODELS FOR REGRESSION Setting this gradient to zero gives 0= ∑N n=1 tnφ(xn)T−wT (N ∑ n=1 φ(xn)φ(xn)T ) . (3.14) So ...

3.1. Linear Basis Function Models 143 Figure 3.2 Geometrical interpretation of the least-squares solution, in anN-dimensional sp ...

144 3. LINEAR MODELS FOR REGRESSION in which the data points are considered one at a time, and the model parameters up- dated af ...

3.1. Linear Basis Function Models 145 q=0. 5 q=1 q=2 q=4 Figure 3.3 Contours of the regularization term in (3.29) for various va ...

146 3. LINEAR MODELS FOR REGRESSION Figure 3.4 Plot of the contours of the unregularized error function (blue) along with the co ...

3.2. The Bias-Variance Decomposition 147 As before, we can maximize this function with respect toW, giving WML= ( ΦTΦ )− 1 ΦTT. ...

148 3. LINEAR MODELS FOR REGRESSION the squared loss function, for which the optimal prediction is given by the conditional expe ...

3.2. The Bias-Variance Decomposition 149 inside the braces, and then expand, we obtain {y(x;D)−ED[y(x;D)] +ED[y(x;D)]−h(x)}^2 = ...

150 3. LINEAR MODELS FOR REGRESSION x t lnλ=2. 6 0 1 −1 0 1 x t 0 1 −1 0 1 x t lnλ=− 0. 31 0 1 −1 0 1 x t 0 1 −1 0 1 x t lnλ=− 2 ...

3.2. The Bias-Variance Decomposition 151 Figure 3.6 Plot of squared bias and variance, together with their sum, correspond- ing ...

152 3. LINEAR MODELS FOR REGRESSION data set leading to large variance. Conversely, a large value ofλpulls the weight parameters ...

3.3. Bayesian Linear Regression 153 Next we compute the posterior distribution, which is proportional to the product of the like ...

154 3. LINEAR MODELS FOR REGRESSION a linear model of the formy(x,w)=w 0 +w 1 x. Because this has just two adap- tive parameters ...

3.3. Bayesian Linear Regression 155 Figure 3.7 Illustration of sequential Bayesian learning for a simple linear model of the for ...

156 3. LINEAR MODELS FOR REGRESSION posterior distribution would become a delta function centred on the true parameter values, s ...

3.3. Bayesian Linear Regression 157 x t 0 1 −1 0 1 x t 0 1 −1 0 1 x t 0 1 −1 0 1 x t 0 1 −1 0 1 Figure 3.8 Examples of the predi ...

158 3. LINEAR MODELS FOR REGRESSION x t 0 1 −1 0 1 x t 0 1 −1 0 1 x t 0 1 −1 0 1 x t 0 1 −1 0 1 Figure 3.9 Plots of the function ...

3.3. Bayesian Linear Regression 159 Figure 3.10 The equivalent ker- nelk(x, x′)for the Gaussian basis functions in Figure 3.1, s ...

160 3. LINEAR MODELS FOR REGRESSION Figure 3.11 Examples of equiva- lent kernels k(x, x′)for x =0 plotted as a function ofx′, co ...