Mathematical Methods for Physics and Engineering : A Comprehensive Guide

STATISTICS

where the quantity denoted byχ^2 is given by the quadratic form

χ^2 (a)=

∑N

i,j=1

[yi−f(xi;a)](N−^1 )ij[yj−f(xj;a)] = (y−f)TN−^1 (y−f).

(31.90)

In the last equality, we have rewritten the expression in matrix notation by

defining the column vectorfwith elementsfi=f(xi;a). We note that in the

(common) special case in which the measurement errorsniareindependent,their

covariance matrix takes the diagonal formN= diag(σ^21 ,σ^22 ,...,σN^2 ), whereσiis

the standard deviation of the measurement errorni. In this case, the expression

(31.90) forχ^2 reduces to

χ^2 (a)=

∑N

i=1

[

yi−f(xi;a)

σi

] 2

.

The least-squares (LS) estimatorsaˆLSof the parameter values are defined as

those that minimise the value ofχ^2 (a); they are usually determined by solving

theMequations

∂χ^2

∂ai

∣

∣

∣

∣

a=aˆLS

=0 fori=1, 2 ,...,M. (31.91)

Clearly, if the measurement errorsniare indeed Gaussian distributed, as assumed

above, then the LS and ML estimators of the parametersacoincide. Because

of its relative simplicity, the method of least squares is often applied to cases in

which theniare not Gaussian distributed. The resulting estimatorsaˆLSarenotthe

ML estimators, and the best that can be said in justification is that the method is

an obviously sensible procedure for parameter estimation that has stood the test

of time.

Finally, we note that the method of least squares is easily extended to the case

in which each measurementyidepends on several variables, which we denote

byxi. For example,yimight represent the temperature measured at the (three-

dimensional) positionxiin a room. In this case, the data is modelled by a

functiony=f(xi;a), and the remainder of the above discussion carries through

unchanged.

31.6.1 Linear least squares

We have so far made no restriction on the form of the functionf(x;a). It so

happens, however, that, for a model in whichf(x;a)isalinearfunction of the

parametersa 1 ,a 2 ,...,aM, one can always obtain analytic expressions for the LS

estimatorsaˆLSand their variances. The general form of this kind of model is

f(x;a)=

∑M

i=1

aihi(x), (31.92)