Mathematical Methods for Physics and Engineering : A Comprehensive Guide

STATISTICS

Setting the expression (31.95) equal to zero ata=aˆ, we find

− 2 RTN−^1 y+2RTN−^1 Raˆ=0.

Provided the matrixRTN−^1 Ris not singular, we may solve this equation foraˆto

obtain

aˆ=(RTN−^1 R)−^1 RTN−^1 y≡Sy, (31.98)

thus defining theM×NmatrixS. It follows that the LS estimatesaˆi,i=1, 2 ,...,M,

are linear functions of the original measurementsyj,j=1, 2 ,...,N.Moreover,

using the error propagation formula (30.141) derived in subsection 30.12.3, we

find that the covariance matrix of the estimatorsaˆiis given by

V≡Cov[ˆai,aˆj]=SNST=(RTN−^1 R)−^1. (31.99)

The two equations (31.98) and (31.99) contain the complete method of least

squares. In particular, we note that, if one calculates the LS estimates using

(31.98) then one has already obtained their covariance matrix (31.99).

Prove result (31.99).

Using the definition ofSgiven in (31.98), the covariance matrix (31.99) becomes

V=SNST

=[(RTN−^1 R)−^1 RTN−^1 ]N[(RTN−^1 R)−^1 RTN−^1 ]T.

Using the result (AB···C)T=CT···BTATfor the transpose of a product of matrices and
noting that, for any non-singular matrix, (A−^1 )T=(AT)−^1 we find

V=(RTN−^1 R)−^1 RTN−^1 N(NT)−^1 R[(RTN−^1 R)T]−^1

=(RTN−^1 R)−^1 RTN−^1 R(RTN−^1 R)−^1

=(RTN−^1 R)−^1 ,

where we have also used the fact thatNis symmetric and soNT=N.

It is worth noting that one may also write the elements of the (inverse)

covariance matrix as

(V−^1 )ij=

1

2

(

∂^2 χ^2

∂ai∂aj

)

a=aˆ

,

which is the same as the Fisher matrix (31.36) in cases where the measurement

errors are Gaussian distributed (and so the log-likelihood is lnL=−χ^2 /2). This

proves,atleastforthiscase,ourearlierstatementthattheLSestimatorsare

minimum-variance. In fact, sincef(x;a) is linear in the parametersa,onecan

writeχ^2 exactlyas

χ^2 (a)=χ^2 (ˆa)+

1

2

∑M

i,j=1

(

∂^2 χ^2

∂ai∂aj

)

a=ˆa

(ai−ˆai)(aj−aˆj),

which is quadratic in the parametersai. Hence the form of the likelihood function