Engineering Optimization: Theory and Practice, Fourth Edition

336 Nonlinear Programming II: Unconstrained Optimization Techniques

Since the gradient vector represents the direction of steepest ascent, the negative

of the gradient vector denotes the direction of steepest descent. Thus any method that

makes use of the gradient vector can be expected to give the minimum point faster

than one that does not make use of the gradient vector. All the descent methods make

use of the gradient vector, either directly or indirectly, in finding the search directions.

Before considering the descent methods of minimization, we prove that the gradient

vector represents the direction of steepest ascent.

Theorem 6.3The gradient vector represents the direction of steepest ascent.

Proof: Consider an arbitary pointXin then-dimensional space. Letfdenote the value

of the objective function at the pointX. Consider a neighboring pointX+dXwith

dX=











dx 1

dx 2

..

.

dxn











(6.57)

wheredx 1 , dx 2 ,... , dxnrepresent the components of the vectordX.The magnitude

of the vectordX,ds, is given by

dXTd X=(ds)^2 =

∑n

i= 1

(dxi)^2 (6.58)

Iff+dfdenotes the value of the objective function atX+dX, the change inf,df,

associated withdXcan be expressed as

df=

∑n

i= 1

∂f

∂xi

dxi= ∇fTdX (6.59)

Ifudenotes the unit vector along the directiondXanddsthe length ofdX, we can

write

dX=uds (6.60)

The rate of change of the function with respect to the step lengthds is given by

Eq. (6.59) as

df

ds

=

∑n

i= 1

∂f

∂xi

dxi

ds

= ∇fT

dX

ds

= ∇fTu (6.61)

The value ofdf/dswill be different for different directions and we are interested in

finding the particular stepdXalong which the value ofdf/dswill be maximum. This

will give the direction of steepest ascent.†By using the definition of the dot product,

†In general, ifdf/ds= ∇fTu > 0 along a vectordX, it is called a direction ofascent, and ifdf/ds <0,

it is called a direction ofdescent.