6.8 Gradient of a Function 337
Eq. (6.61) can be rewritten as
df
ds
= ||∇f|| ||u||cosθ (6.62)
where||∇f||and||u||denote the lengths of the vectors∇fandu, respectively, andθ
indicates the angle between the vectors∇f andu. It can be seen thatdf/dswill be
maximum whenθ= 0 ◦and minimum whenθ= 180 ◦. This indicates that the function
value increases at a maximum rate in the direction of the gradient (i.e., whenuis
along∇f).
Theorem 6.4The maximum rate of change off at any pointXis equal to the mag-
nitude of the gradient vector at the same point.
Proof: The rate of change of the functionf with respect to the step lengthsalong a
directionuis given by Eq. (6.62). Sincedf/dsis maximum whenθ= 0 ◦and uis a
unit vector, Eq. (6.62) gives
(
df
ds
)∣
∣
∣
∣
max
= ||∇ f||
which proves the theorem.
6.8.1 Evaluation of the Gradient
The evaluation of the gradient requires the computation of the partial derivatives∂f/∂xi,
i= 1 , 2 ,... , n. There are three situations where the evaluation of the gradient poses
certain problems:
1.The function is differentiable at all the points, but the calculation of the com-
ponents of the gradient,∂f/∂xi, is either impractical or impossible.
2 .The expressions for the partial derivatives∂f/∂xi can be derived, but they
require large computational time for evaluation.
3.The gradient∇fis not defined at all the points.
In the first case, we can use the forward finite-difference formula
∂f
∂xi
∣
∣
∣
∣
Xm
≃
f(Xm+ xiui) −f(Xm)
xi
, i= 1 , 2 ,... , n (6.63)
to approximate the partial derivative∂f/∂xiatXm. If the function value at the base
pointXmis known, this formula requires one additional function evaluation to find
(∂f/∂xi)|Xm. Thus it requiresnadditional function evaluations to evaluate the approxi-
mate gradient∇f|Xm. For better results we can use the central finite difference formula
to find the approximate partial derivative∂f/∂xi|Xm:
∂f
∂xi
∣
∣
∣
∣
Xm
≃
f(Xm + xtui) −f(Xm− xiui)
2
xi
, i= 1 , 2 ,... , n (6.64)
