2 Classical Optimization Techniques
2.1 Introduction
The classical methods of optimization are useful in finding the optimum solution of
continuous and differentiable functions. These methods are analytical and make use
of the techniques of differential calculus in locating the optimum points. Since some
of the practical problems involve objective functions that are not continuous and/or
differentiable, the classical optimization techniques have limited scope in practical
applications. However, a study of the calculus methods of optimization forms a basis for
developing most of the numerical techniques of optimization presented in subsequent
chapters. In this chapter we present the necessary and sufficient conditions in locating
the optimum solution of a single-variable function, a multivariable function with no
constraints, and a multivariable function with equality and inequality constraints.
2.2 Single-Variable Optimization
A function of one variablef (x)is said to have arelativeorlocal minimumatx=
x∗if f(x∗) ≤f(x∗+ h)for all sufficiently small positive and negative values ofh.
Similarly, a pointx∗is called a relativeorlocal maximumiff (x∗) ≥f(x∗+ h)for
all values ofhsufficiently close to zero. A functionf (x)is said to have aglobal
orabsolute minimumatx∗if f(x∗) ≤f(x)for allx, and not just for allxclose to
x∗, in the domain over whichf(x)is defined. Similarly, a pointx∗will be a global
maximum off (x) iff (x∗) ≥f(x)for allx in the domain. Figure 2.1 shows the
difference between the local and global optimum points.
Asingle-variable optimization problemis one in which the value ofx=x∗is to be
found in the interval [a, b] such thatx∗minimizesf(x). The following two theorems
provide the necessary and sufficient conditions for the relative minimum of a function
of a single variable.
Theorem 2.1 Necessary Condition If a functionf (x)is defined in the intervala≤
x≤band has a relative minimum atx=x∗, where a< x∗< b ,and if the derivative
df (x)/dx=f′(x) exists as a finite number atx=x∗, thenf′(x∗) = 0.
Proof: It is given that
f′(x∗) =lim
h→ 0
f (x∗+ h)−f (x∗)
h
(2.1)
Engineering Optimization: Theory and Practice, Fourth Edition Singiresu S. Rao 63
Copyright © 2009 by John Wiley & Sons, Inc.
