Optimization Techniques.pdf

Optimization Techniques
Anand J Kulkarni
PhD, MASc, BEng, DME
Professor & Associate Director
Institute of Artificial Intelligence, MIT World Peace University, Pune,
Bharat (India)
Email: anand.j.kulkarni@mitwpu.edu.in; kulk0003@ntu.edu.sg

What is optimization about?
• Extreme states (i.e. minimum and maximum states out of many or
possibly infinitely many)
Ex. Natural (physical) stable equilibrium state is generally a
‘minimum potential energy’ state.
• Human activities: to do the best in some sense
• set a record in a race (shortest/minimum time, etc.)
• retail business (maximize the profit, etc.)
• construction projects (minimize cost, time, etc.)
• power generator design (maximize efficiency, minimize weight,
etc.)
• Best job out of several choices
2

• Concept of (Optimization) minimization and
maximization to achieve the best possible
outcome is a positive and intrinsic human
nature.
• Study of optimization:
• Create optimum designs of products,
processes, systems, etc.
3

• Real world issues:
• Requirements and constraints imposed on
products, systems, processes, etc.
• Creating feasible design (solution)
• Creating a best possible design (solution)
• “Design optimization”: highly complex, conflicting
constraints and considerations, etc.
4

What is an Optimal Design (Tai et al.)
• In the engineering design of any component,
device, process or system the optimal design
can be defined as the one that is feasible and
the best according to some quantitative
measure of effectiveness.
5

Importance of Optimization
• Greater concern about the limited energy,
material, economic sources, etc.
• Heightened environmental/ecological
considerations
• Increased technological and market
competition
6

A Simple Example
• 5 X 7 metal sheet
• can take different values between 0 and 2.5
• Infinite box designs (solutions)
• Aim: Biggest box volume (Maximization)
x
7

A Simple Example
( ) ( )( ) 3 2
5 2 7 2 4 24 35 , 0 2.5
f x x x x x x x x
= − − = − +  
8

9
-10
0
10
20
30
40
50
60
70
80
0 1 2 3 4 5 6

Sensor Network Coverage Problem
12
Kulkarni, A.J., Tai, K., Abraham, A.: “Probability Collectives: A Distributed Multi-Agent System
Approach for Optimization”, Intelligent Systems Reference Library, 86 (2015) Springer

13
( )
,
l l
x y
( )
,
u u
x y
( )
,
u l
x y
s
r
s
r
Sensor 3
,1
c
A
,2
c
A
Square
FoI
( )
1 1
,
x y
Enclosing
Rectangle
A
( )
,
l u
x y
( )
2 2
,
x y
,3
c
A
( )
3 3
,
x y
s
r
Sensor 2
Sensor 1

14
( )
,
l l
x y
( )
,
u u
x y
( )
,
u l
x y
s
r
s
r
Sensor 3
,1
c
A
,2
c
A
Square
FoI
( )
1 1
,
x y
Enclosing
Rectangle
A
( )
,
l u
x y
( )
2 2
,
x y
,3
c
A
( )
3 3
,
x y
s
r
Sensor 2
Sensor 1

A Simple Example
• 5 X 7 metal sheet
• can take different values between 0 and 2.5
• Infinite box designs (solutions)
• Aim: Biggest box volume (Maximization)
x
16

A Simple Example
• Setting the obtain stationary points
or
( )
'
0
f x =
0.96
x = 3.04
x =
( ) ( )
0.96 15.02 ; 3.04 3.02
f f
= = −
( ) ( )( )
( )
( )
3 2
' 2
2
''
2
5 2 7 2 4 24 35 , 0 2.5
12 48 35
24 48
f x x x x x x x x
df
f x x x
dx
d f
f x x
dx
= − − = − +  
= = − +
= = −
17
-10
0
10
20
30
40
50
60
70
80
0 1 2 3 4 5 6

18
-10
0
10
20
30
40
50
60
70
80
0 1 2 3 4 5 6

Illustration 1
19
( ) 3 2
5 2 3
f x x x x
= + −
Find Maximum and Minimum of
( )' 2
15 4 3
f x x x
= + −
First order derivative
3 1
5 3
x and x
= − =
First order derivative equated to zero,
i.e. locate variable values at which
slope becomes zero
Second order derivative ( )''
30 4
f x x
= +
Second order derivative value at ( ) ( )
'' ''
3 1
5 3
14 14
and
f x f x
−
= − =

Illustration 1
Maximum at
Minimum at
20
3
5
x = −
-4
-2
0
2
4
6
8
10
-1.5 -1 -0.5 0 0.5 1 1.5
1
3
x =

Illustration 2
• A ball is thrown in the air. Its height at any
time t is given by ℎ = 3 + 14𝑡 − 5𝑡2
. What is
the maximum height?
21

22
𝑑ℎ
𝑑𝑡
= 14 − 10𝑡 = 0 , i. e. locating point at which slope is zero
0
2
4
6
8
10
12
14
0 0.5 1 1.5 2 2.5 3 3.5
𝑡 = 1.4
ℎ𝑒𝑖𝑔ℎ𝑡 ℎ = 12.8
𝑑2ℎ
𝑑𝑡2
= −10 , What does this mean?
This means the slope is continually getting smaller (−10): traveling from left to right the
slope starts out positive (the function rises), goes through zero (the flat point), and then the
slope becomes negative (the function falls)
ℎ = 3 + 14𝑡 − 5𝑡2

• 3D view of 2D optimization
25

Basic Definitions
• A design problem is characterized by a set of
design variables (decision variables)
• Single Variable
• Multi-Variable
where
( ) ( )
( )
2
min 2 log
f x x x
= +
( ) ( )
( )
( ) ( )
( )
5
1 2 1 2
5
1 2
min , 2 log
2 log
f x x x x
f x x
= +
= +
X
( )
1 2
,
x x
=
X
26

Basic Definitions
• Design variables
• Continuous (any value between a specific interval)
• Discrete (the value from a set of distinct numerical
values)
• Ex. Integer values, (1, 4.2, 6 11, 12.9, 25.007), binary (0,
1), etc.
• Combinatorial Optimization Problem
• Mixed (discrete & continuous) variables
27

30
http://mathgifs.blogspot.in/2014/03/the-traveling-salesman.html

Basic Definitions
• Objective Function:
• Criterion by which the performance/effectiveness
of the solution is measured
• Also referred to as ‘Cost Function’.
Multi-objective
Problem
( ) ( )( )
( ) ( )
( )
( ) ( )
( )
5
1 2 1 2
5
1 2
5 2 7 2
, 2 log
2 log
f x x x x
f x x x x
f x x
= − −
= +
= +
X
31

Basic Definitions
• Unconstrained Optimization Problems
• No restrictions (Constraints) imposed on
design variables
• Constrained Optimization Problems
• Restrictions (constraints) are imposed on
design variables and the final solution
should satisfy these constraints, i.e. the final
solution should at least be feasible.
• The ‘best’ solution comes further
32

Basic Definitions
• Depending on physical nature of the problem:
‘Optimal Control Problem’
• Decomposing the complex problem into a
number of simpler sub-problems
• Linear Programming (LP): If the objective and
constraints are linear
• Non-Linear programming (NLP): If any of it is non-
linear
33

General Problem Statement
• Side constraints
( )
( )
( )
:
0 , 1,...,
0 , 1,...,
, 1,...,
j
k
i
Minimize f
Subject to
g j m
h k l
x i n
 =
= =
=
X
X
X
l u
i i i
x x x
 
34

Active/Inactive/Violated Constraints
• Inequality Constraints
35
( ) ( )
1
2
3
4
,
12 5 3000
10 14 4000
50 50
50 50
f f B R
g B R
g B R
g B B
g R R
=
= + 
= + 
=  → −  −
=  → −  −
X

• The set of points at which an inequality
constraint is active forms a constraint
boundary which separates the feasible region
points from the infeasible region points.
36

• An inequality constraint is said to be
violated at a point , if it is not satisfied there
i.e. .
• If is strictly satisfied i.e. . Then it is
said to be inactive at the point .
• If is satisfied at equality i.e. . Then
it is said to be active at the point .
37
j
g
( )
( )
0
j
g 
X
j
g ( )
( )
0
j
g 
X
X
X
( )
( )
0
j
g =
X
j
g
X

38
1 2
2
3
4
5
240000000
10 0
450000
2 0
2
2 0
0
0
g
bd
g
bd
g d b
g b
g d
= − 
= − 
= − 
= − 
= − 

• Based on these concepts, equality constraints
can only be either active i.e. or
violated i.e. at any point .
39
( )
( )
0
j
h =
X
( )
( )
0
j
h 
X X

• Equality & inequality constraints
40
( ) ( )
1 2
1 1 2
2 1 2
3 1 2
1 1 1
2 2 2
,
4 2 12
1
2 4
0 0
0 0
f f x x
h x x
h x x
h x x
g x x
g x x
=
= + =
= − + =
= + =
=  → − 
=  → − 
X
Design Space
Feasible Region
Infeasible Region

Practical Example
• A company manufactures two machines, A and B. Using available
resources, either 28 A or 14 B can be manufactured daily. The sales
department can sell up to 14 A machines or 24 B machines. The
shipping facility can handle no more than 16 machines per day. The
company makes a profit of $400 on each A machine and $600 on
each B machine. How many A and B machines should the company
manufacture every day to maximize its profit?
43

44
𝑥1 = 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑀𝑎𝑐ℎ𝑖𝑛𝑒𝑠 𝑡𝑦𝑝𝑒 𝐴 𝑚𝑎𝑛𝑢𝑓𝑎𝑐𝑡𝑢𝑟𝑒𝑑 𝑒𝑎𝑐ℎ 𝑑𝑎𝑦
𝑥2 = 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑀𝑎𝑐ℎ𝑖𝑛𝑒𝑠 𝑡𝑦𝑝𝑒 𝐵 𝑚𝑎𝑛𝑢𝑓𝑎𝑐𝑡𝑢𝑟𝑒𝑑 𝑒𝑎𝑐ℎ 𝑑𝑎𝑦
𝑃𝑟𝑜𝑓𝑖𝑡 𝑃 = 400𝑥1 + 600𝑥2
𝑆ℎ𝑖𝑝𝑝𝑖𝑛𝑔 𝑎𝑛𝑑 ℎ𝑎𝑛𝑑𝑙𝑖𝑛𝑔 𝐶𝑜𝑛𝑠𝑡𝑟𝑎𝑖𝑛𝑡𝑠 𝑥1 + 𝑥2 ≤ 16
𝑀𝑎𝑛𝑢𝑓𝑎𝑐𝑡𝑢𝑟𝑖𝑛𝑔 𝐶𝑜𝑛𝑠𝑡𝑟𝑎𝑖𝑛𝑡
𝑥1
28
+
𝑥2
14
≤ 1
𝐿𝑖𝑚𝑖𝑡𝑎𝑡𝑖𝑜𝑛 𝑜𝑛 𝑆𝑎𝑙𝑒𝑠 𝐷𝑒𝑝𝑎𝑟𝑡𝑚𝑒𝑛𝑡
𝑥1
14
+
𝑥2
24
≤ 1
𝑥1 , 𝑥2 ≥ 0

Convexity
• A set of points is a convex set, if for any two
points in the set, the entire straight line
segment joining these two points is also in the
set.
45

Convexity
• A function f(X) is convex if it is defined over a
convex set and for any two points of the graph
f(X), the straight line segment joining these
two points lies entirely above or on the graph.
46

Local and Global Optimum
• An objective function is at its local minimum
at the point if for all feasible
within its small neighborhood of
47
( ) ( )
( )
*
f f

X X
*
X
f
X
*
X

Local and Global Optimum
• An objective is at its global minimum at the
point if for all feasible .
48
( ) ( )
( )
*
f f

X X
*
X
f
X

Gradient Vector and Hessian Matrix
• The techniques of finding the optimum point
largely depend on ‘calculus’.
• The usual assumption of continuity of the
objective and constraint functions is required (at
least to first order).
• The ‘derivatives’ of these functions wrt each
variable are important as they indicate the rate of
change of the function wrt these variables and
also respective stationary points.
49

• Gradient Vector: first order derivative
Hessian Matrix: second order derivative
50
( ) ( ) ( ) ( )
1 2
...
n
f f f
f
x x x
 
  
 =  
  
 
X X X
X
( )
( ) ( ) ( )
( ) ( ) ( )
( ) ( ) ( )
2 2 2
2
1 2 1
1
2 2 2
2
2
2 1 2
2
2 2 2
2
1 2
...
...
n
n
n n n
f f f
x x x x
x
f f f
x x x x
x
f
f f f
x x x x x
 
  
 
   

 
 
  
 
   

 =
 
 
 
  
 
    
 
 
X X X
X X X
X
X X X

• The gradient and Hessian of the objective
function and constraint functions provides
sensitivity analysis, i.e. they indicate how
sensitive the functions are towards changes of
the design variables.
• Example:
51
( ) 3 2 2
1 1 3 1 2
3 5 9
f x x x x x
= + + +
X

General Optimization Unconstrained
• Analytical methods are too cumbersome to be
used for non-linear optimization problems.
Hence numerical approach required.
• Reasons: Complexity grows as number of
design variables and/or constraints grows.
• Implicit constraints
52

53

1. Estimate a reasonable starting design
where is iteration counter
2. Compute the search direction in the
design space.
3. Check for the convergence.
4. Calculate the step size in the direction
5. Update the design
6. And set and go to Step 2.
54
( )
0
X
k
( )
k
d
k
 ( )
k
d
1
k k
= +
( ) ( ) ( )
1
k k k
k

+
= +
X X d Step to move
further

• Desirable search direction = descent direction
• Taylor’s Expansion
• where
55
( )
( ) ( )
( )
1
k k
f f
+

X X
( ) ( )
( ) ( )
( )
k k k
k
f f

+ 
X d X
( ) ( ) ( )
( )
( ) ( )
( )
( ) ( )
0
k k k k
k
k k
f f

+  
 
X c d X
c d
( ) ( )
( )
k k
f
= 
c X
( )
1
k−
X
( )
k
X
( )
1
k+
X
( )
k
d
( )
1
k
 +
d

• Example:
• Check the direction at point is a
descent direction for .
57
( ) 1 2
2 2
1 1 2 2 1
2 2 x x
f x x x x x e +
= − + − +
X
( )
1,2
=
d ( )
0,0
( )
f X

Single Variable Optimization
Unconstrained
• Basic approach behind any numerical
technique is to divide the problem into two
phases.
Phase 1: Initial bracketing or bounding of the
minimum point
Phase 2: approximation of the minimum point
58

Initial bracketing or bounding of the
minimum point
• A small value of is specified.
• Function values are evaluated at a sequence
of points along a uniform grid with a constant
search interval , i.e. at points
• The function will be in general decreasing
until it starts increasing, indicating that the
minimum is surpassed, i.e.
59

 0, ,2 ,3 ,...
x   
=
( )
f x
*
x
( )
( ) ( )
( ) ( )
( )
1 , 0
1
f s f s s is an integer
and f s f s
 
 
−  
 +

minimum point
• Equal interval search method
60
0  2 ( )
1
s 
− s ( )
1
s 
+ x
( )
f x
*
x

minimum point
• Whenever this condition arises at any integer
the minimum is bracketed in the interval
61
s
( ) ( )
*
1 1
s x s
 
−   +

minimum point
• Issues:
the efficiency depends on the number of
function evaluations which depends on the
value of .
- too small- large function evaluations
- too large- resulting bracket (interval) is
wide and may need more computations in
the second phase of locating approximate
minimum point
62


minimum point
• Variable interval search method
To improve upon the previous search method
- initial is specified
- subsequent search intervals are incremented
by a fixed ratio .
i.e. the function will be evaluated at a
series of points
63

a
( )
f x
( ) ( ) ( )
2 2 3
0, , 1 , 1 , 1 ,...
x a a a a a a
   
= + + + + + +

minimum point
• Therefore
where is the iteration number
• Similar to the equal interval search the
minimum point is bounded
64
( ) ( )
1
, 0,1,2,3,...
s s s
x x a s

+
= + =
s
( )
( ) ( )
( )
( )
( ) ( )
( )
( ) ( )
1
1 1
*
1
s s
s s
s s
f x f x
x x x
f x f x
−
− +
+

 
  

 


minimum point
• One good ratio to use is ‘Golden Ratio’
• The ratio is based on the ‘Golden Section Method’ which is
one of the methods for the second phase, i.e. to locate the
approximate minimum.
65

Approximation of the Minimum
• Reduce the interval to a small enough size so that the
minimum will be known with an acceptable
precision.
• Golden Section Method and Polynomial
Approximation (Polynomial Interpolation)
66
*
x

Approximation of the Minimum:
Golden Section Method
Evaluate the two sectioning points and
within the interval such that
The size of the interval is repeatedly reduced
by a fraction each time by
discarding portion either or …
67
1
x 2
x
1 2
l u
x x x x
  
( ) ( )
( ) ( )
1
2
1 2
2 1
l u
l u
x a x a x
x a x a x
= − + −
= − + −
( )
2 0.38197
a
− 
1
l
x x
 2 u
x x


… based on:
and
68
( ) ( )
1 2
*
2
l
f x f x
x x x

 
2
2 1
1 ?
u
l l
x x
x x
x x
x
=
=
=
=
( ) ( )
1 2
*
1 u
f x f x
x x x

 
1
1 2
2 ?
l
u u
x x
x x
x x
x
=
=
=
=
( )
f x
( )
f x
l
x 1
x 2
x u
x
l
x 1
x 2
x u
x
x
x

• Convergence Criteria
– Precision limit
‘current interval’ as compared to the ‘original
interval’
Finally,
69
( ) ( )
0 0
, 0.00001
u l
u l
x x
x x
 
−
 =
−
*
2
l u
x x
x
+
=

Why Golden Ratio?
• Do not need to recalculate the function values
at and in every iteration but either of
both only.
70
1
x 2
x

Example: Golden Section Method
• Example: Golden Section Method
• Bounded by
71
( ) 2 4 x
f x x e
= − +
0.5 2.618
l u
x and x
= =

72
( ) ( )
( )
( ) ( )
( )
( )
( ) ( )
( ) ( ) ( )
( )
( )
( ) ( )
( ) ( ) ( )
( )
( )
( ) ( )
( )
0 0
0 0
0 0 0 0
1 1
0 0 0 0
2 2
0 0
1 2
0.5 1.649
2.618 5.237
1 2 1.309 0.466
2 1 1.809 0.868
l l
u u
l u
l u
x f x
x f x
x a x a x f x
x a x a x f x
Now f x f x
= → =
= → =
= − + − = → =
= − + − = → =

( )
f x
( )
0
l
x ( )
0
1
x ( )
0
2
x ( )
0
u
x

73
( ) ( ) ( )
( ) ( )
( )
( ) ( ) ( )
( ) ( )
( )
( ) ( ) ( )
( ) ( )
( )
( )
( )
( ) ( )
( ) ( ) ( )
( )
1 0 1 0
1 0 1 0
2 1 2 1
1 0 1 0
2 2
1
1
1 1 1 1
1 1
0.5 1.649
1.309 0.466
1.809 0.868
?
1 2 1.0 0.718
l l l l
u u
l u
x x f x f x
x x f x f x
x x f x f x
x
x a x a x f x
= = → = =
= = → = =
= = → = =
=
= − + − = → =
( )
f x
( )
1
l
x ( )
1
1
x
( )
1
2
x ( )
1
u
x
( )
( ) ( )
( )
1 1
1 2
Now f x f x


74
( ) ( ) ( )
( ) ( )
( )
( ) ( ) ( )
( ) ( )
( )
( ) ( ) ( )
( ) ( )
( )
( )
( )
( ) ( )
( ) ( ) ( )
( )
2 1 2 1
1 1
2 1 2 1
1 2 1 2
2 1 2 1
2
2
2 2 2 2
2 2
1.0 0.718
1.309 0.466
1.809 0.868
?
2 1 1.5 0.482
...
l l
u u u u
l u
x x f x f x
x x f x f x
x x f x f x
x
x a x a x f x
and so on
= = → = =
= = → = =
= = → = =
=
= − + − = → =

Interval Halving Method
76
◼ Algorithm
◼ Step 1: Let , xm = (xu+xl)/2 find f(xm)
◼ Step 2: Set x1 = xl + (xu-xl)/4, x2 = xu - (xu-xl)/4.
◼ Step 3: If f(x1)<f(xm), then xu =xm, xm= x1 go to Step 1.
◼ elseif f(x1)>f(xm), continue to Step 4
◼ Step 4: If f(x2)<f(xm), then xl=xm, xm=x2 go to Step 1.
If f(x2)>f(xm), then xl=x1, xu=x2, go to Step 1.

Polynomial Approximation
• Golden Section method may require more
number of iterations and function evaluations
as compared to polynomial approximation
method
• Pass a polynomial curve through a
certain number of points of the function .
• And the minimum of this known curve will be
a good estimate of the minimum point of the
actual function .
77
( )
f x
( )
f x
( )
ˆ
f x

• Generally the curve is a ‘second order’
polynomial
• 3 common points are required as 3 unknown
coefficients are to be found out.
78
( ) 2
0 1 2
ˆ
f x a a x a x
= + +
0 1 2
,
a a and a

79
l
x u
x x
( )
f x
im
x

are already known, So…
Solve these simultaneous equations to find
.
80
( ) ( ) ( )
, ,
l im u l im u
x x and x and f x f x and f x
( ) ( )
( ) ( )
( ) ( )
2
0 1 2
2
0 1 2
2
0 1 2
ˆ
ˆ
ˆ
l l l l
im im im im
u u u u
f x a a x a x f x
f x a a x a x f x
f x a a x a x f x
= + + =
= + + =
= + + =
0 1 2
,
a a and a

• This will give us the complete equation of the
polynomial curve .
• Find out
• Equate and compute ….
and
81
( ) 2
0 1 2
ˆ
f x a a x a x
= + +
( ) ( )
' ''
ˆ ˆ
f x and f x
( )
'
ˆ 0
f x = *
x
* 1
2
2
a
x
a
= −
( )
''
2
ˆ 2 0
f x a
= 

• The more accurate results could be found out
by further refining the approximation, i.e. …
• Now are available, so by
comparing respective
discard any of the above points and …
continue the procedure…
82
*
, ,
l im u
x x x and x
( ) ( ) ( ) ( )
*
, ,
l im u
f x f x f x and f x

• ….till convergence, i.e. until the two successive
estimates of the minimum points are
sufficiently close to each other, i.e.
• Higher order polynomial curve is also possible.
83
( )
* 1 *
r r
x x 
+
− 

Example: Polynomial Approximation
• Example:
• Bounded by
• can also be found by an average
84
( ) 2 4 x
f x x e
= − +
0.5, 1.309 2.618
l im u
x x and x
= = =
im
x

85
( ) ( )
( )
( ) ( )
( )
( ) ( )
( )
( ) ( ) ( )
0 0
0 0
0 0
0 0 0
0 1 2
0.5 1.649
1.309 0.466
2.618 5.237
, 5.821 2.410
l l
im im
u u
x f x
x f x
x f x
find a a and a
= → =
= → =
= → =
= − =
( )
( )
( )
( ) ( )
* 0
* 0
5.821
1.208
2 2.410
1.208 0.515
x
f x f
−
= =
= =

86
( ) ( )
( )
( ) ( )
( )
( ) ( )
( )
( ) ( )
( )
0 0
0 0
0 0
* 0 * 0
0.5 1.649
1.309 0.466
2.618 5.237
1.208 0.515
l l
im im
u u
x f x
x f x
x f x
x f x
= → =
= → =
= → =
= → =
( ) ( ) ( )
( ) ( )
( )
* 0 0 * 0 0
im im
x x and f x f x
 

87
( ) ( ) ( )
( ) ( )
( )
( ) ( ) ( )
( ) ( )
( )
( ) ( ) ( )
( ) ( )
( )
1 * 0 1 * 0
1 0 1 0
1 0 1 0
1.208 0.515
1.309 0.466
2.618 5.237
...
l l
im im im im
u u u u
x x f x f x
x x f x f x
x x f x f x
and so on
= = → = =
= = → = =
= = → = =

Examples
88
( )
( )
4 3 2
0.25 0.2 8 4
x
f x x x x x
f x e x
= + + − +
= +

Newton Raphson Method
Step 1 Choose initial Guess 𝑥(0)
and convergence parameter ε.
Step 2 Compute 𝑓′ 𝑥 𝑘 𝑎𝑛𝑑 𝑓′′(𝑥(𝑘)).
If |𝑓′
𝑥 𝑘
| < ε Then Terminate
Step 3 Calculate 𝑥(𝑘+1)
= 𝑥(𝑘)
−
𝑓′ 𝑥 𝑘
𝑓′′ 𝑥 𝑘 . Compute 𝑓′
(𝑥(𝑘+1)
).
Step 4 If |𝑓′
𝑥 𝑘+1
| < ε Then Terminate
Else set 𝑘 = 𝑘 + 1 and go to Step 2.
Solve:
𝒇 𝒙 = 𝒙𝟐 +
𝟓𝟒
𝒙
Choose 𝒙 𝟏
= 𝟏 and ε = 0.001
90

91
𝑓 𝑥 = 3𝑥3
− 10𝑥2
− 56𝑥
𝑓′ 𝑥 = 9𝑥2
− 20𝑥 − 56
𝑓′′ 𝑥 = 18𝑥 − 20
−4 ≤ 𝑥 ≤ 6

Bisection Method
Using the derivative information, the minimum is supposed to be bracketed in
the interval 𝑎, 𝑏 if two conditions 𝑓′ 𝑎 < 0 and 𝑓′ 𝑏 > 0 along with the
convergence condition are satisfied .
Step 1 Choose two points 𝑎 and 𝑏 such that 𝑓′
𝑎 < 0 and 𝑓′
𝑏 > 0,
convergence parameter ε.
Set 𝑥1 = 𝑎 and 𝑥2 = 𝑏.
Step 2 Calculate 𝑧 = (𝑥1 + 𝑥2)/2 and evaluate (calculate) 𝑓′ 𝑧 .
Step 3 If |𝑓′ 𝑧 | ≤ ε Then Terminate
Else if 𝑓′ 𝑧 < 0 set 𝑥1 = 𝑧 and go to Step 2
Else if 𝑓′ 𝑧 > 0 set 𝑥2 = 𝑧 and go to Step 2
Solve:
𝒇 𝒙 = 𝒙𝟐 +
𝟓𝟒
𝒙
Choose 𝒂 = 𝟐 and 𝒃 = 𝟓 and ε = 0.001
92

Secant Method
In this method, both the magnitude and sign of the derivatives at two points
under consideration are used. The derivative of the function is assumed to
vary linearly between two chosen boundary points. As the derivative at these
two points changes sign, the point with zero derivative most probably lies
between these two points.
Step 1 Choose two points 𝑎 and 𝑏 such that 𝑓′ 𝑎 < 0 and 𝑓′ 𝑏 > 0,
convergence parameter ε.
Set 𝑥1 = 𝑎 and 𝑥2 = 𝑏.
Step 2 Calculate 𝑧 = 𝑥2 −
𝑓′(𝑥2)
(𝑓′ 𝑥2 −𝑓′(𝑥1))/(𝑥2−𝑥1)
and evaluate (calculate)
𝑓′
𝑧 .
Step 3 If |𝑓′
𝑧 | ≤ ε Then Terminate
Else if 𝑓′
𝑧 < 0 set 𝑥1 = 𝑧 and go to Step 2
Else if 𝑓′
𝑧 > 0 set 𝑥2 = 𝑧 and go to Step 2
93

Secant Method
Solve:
𝑓 𝑥 = 𝑥2
+
54
𝑥
Choose 𝑎 = 2 and 𝑏 = 5 and ε = 0.001
94

Multivariable Optimization
• Steepest Descent Method
1. Estimate a starting design , select a
convergence parameter .
2. Calculate the gradient of the function at
point , i.e.
3. If , stop!!! … And accept the current
design and associated as the final
solution.
96
( ) ( )
( )
( )
( ) ( )
( ) ( )
( )
1 2
...
k k k
k k
n
f f f
f
x x x
 
  
 
=  =  
  
 
 
X X X
c X
( )
k
X
0.0001
 
( )
f X
( )
k
X
c
( )
k


c
( )
k
X ( )
f X

Multivariable Optimization
4. Based on the desirable search direction
condition, , decide the search
direction at the current point to be
.
5. Calculate/pre-specify the step size .
6. Update the new design point
and calculate the function value at , i.e.
.
7.Go to step 2. 97
( ) ( )
0
k k
 
c d
( )
k
X
( ) ( )
k k
= −
d c

( ) ( ) ( )
1
k k k

+
= +
X X d
( )
( )
1
k
f +
X
( )
1
k+
X

Limitations of Steepest Descent
Method
• Limitations of Steepest Descent Method:
1. Large number of iterations required (zigzag??) and
process becomes quite slow to converge.
2. Every iteration is independent of the previous
iteration(s), so inefficient.
3. Only first order information (gradient) is
used, i.e. no Hessian is used so no information
about the condition number is exploited.
4. Process is faster initially but slows down when
reaches closer to local minimum.
5. Local minimum is guaranteed and not the global
minimum.
99

Example
100
( ) ( )
( )
( )
( )
( )
( )
( ) ( )
( ) ( ) ( )
( )
2 2
1 2 1 2 1 2
0
0 0
0
1 2 2 1
1 2
0
, 2
1,0 0.05
1. 1,0
2. 2 2 ,2 2 2, 2
3. 2 2
Minimize f x x f x x x x
starting point and
Starting design
f f
x x x x
x x


= = + −

=
 
 
 
= = − − = −
 
 
 
 
= 
X
X
X X
c
c

Example
101
( ) ( )
( )
( ) ( ) ( )
( )
( ) ( )
( )
( )
( )
( )
( )
( )
( )
( ) ( )
( ) ( ) ( )
( )
0 0
1 0 0
1 1
1 2
1
1
1 1
1
1 2 2 1
1 2
1
4. 2,2
5.
. . 1 0.25 2 0.5; 0 0.25 2 0.5
. . 0.5,0.5
0.5
2 2 ,2 2 0,0
0 ...
Set
Calculate the new design point:
i e x x
i e
and f
f f
x x x x
x x


= − = −
= +
= + − = = + =
=
= −
 
 
 
= = − − =
 
 
 
 
= → 
d c
X X d
X
X
X X
c
c

Example
102
( ) ( )
( ) ( )
( )
( )
( ) ( ) ( ) ( )
( )
( ) ( )
( )
1 1 1
1 2
1
1 1 1
1 2 1 2
1
!!! ,
. . : , ,
final final final
final
stop and accept the current variable values x x
and associated f as final solution
i e final solution x x x x
f f
=
= = =
=
X
X
X X
X X

Example
104
( ) ( )
( )
2 2 2
1 2 3 1 2 3 1 2 2 3
, , 2 2 2 2
2,4,10 0.005
Minimize f x x x f x x x x x x x
starting point and 
= = + + + +

X

Modified Steepest Descent Method:
Conjugate Gradient Method
• Every iteration solution is dependent on the
previous iteration(s), so more efficient than the
steepest descent method. Uses the deflected
direction based on the previous iteration
direction.
• So replace with
which satisfies the desirable descent direction
condition.
105
( ) ( )
k k
= −
d c
( ) ( ) ( ) ( )
( ) ( ) ( )
( )
1
2
1
k k k k
k k k
where


−
−
= − +
=
d c d
c c

Newton Method
• With the steepest descent method, only first-
order derivative information is used.
• If second-order derivatives were available, we
could use them to represent the cost surface
more accurately, and a better search direction
could be found, i.e. use the Hessian
• Better Convergence (‘quadratic rate of
convergence’) if the minimum is located in a
the neighborhood of certain radius.
106

Newton Method
• The quadratic expression for the change in
design at some point can be
represented using Taylor Series as:
• If is positive definite (eigenvalues and
determinant are positive) then the global
minimum is guaranteed.
107
X X
( ) ( ) 0.5
T T
f f
+  = +  +  
X X X c X X H X
H

Newton Method
• The optimality condition
• Reduces the Taylor’s Expansion to
108
( )
'
0
f =
X
( ) ( ) ( ) ( )
1
1 0 0 0
0
...
assuming is non-singular/positive definite
so
 
−
+  =
 = −
= +   +
c H X
H
X H c
X X X X d

Newton Method
1. Estimate a starting design point
2. Select the convergence parameter
3. Calculate the gradient of the function
at point , i.e.
solution.
109
( )
k
X
0.001
 =
( ) ( )
( )
( )
( ) ( )
( ) ( )
( )
1 2
...
k k k
k k
n
f f f
f
x x x
 
  
 
=  =  
  
 
 
X X X
c X
( )
f X
( )
k
X
( )
k
c
( )
k


c
( )
k
X ( )
f X

Newton Method
5. Calculate Hessian at point .
6. Calculate the ‘search direction’:
7. Check whether the search direction is ‘desirable’
by checking its positive definiteness or the
following condition
110
( )
k
H ( )
k
X
( ) ( ) ( )
1
k k k
−
 
= −  
d H c
( ) ( ) ( ) ( ) ( )
1
0 0
T T
k k k k k
−
 
     
    − 
 
     
 
c d c H c

8. Update the design
111
( ) ( ) ( )
1
k k k

+
= +
X X d

Example
112
( )
( )
( )
2 2
1 1 2 2
0
3 2 2 7
5,10 , 0.0001
f x x x x

= + + +
= =
X
X

Example
113
( )
( )
 
( )
   
( )
( )
( )
2 2
1 1 2 2
0
0
1 2 1 2
0 2 2
0
3 2 2 7
1. 5 10
2. 6 2 2 4 50 50
50 50 50 2 ...
6 2
3.
2 4
4.
...
f x x x x
x x x x
convergence condition not satisfied
Check the condition for desirable search direction
eigenvalues positive definiteness or

= + + +
=
= + + =
= + = 
 
=  
 
X
X
c
c
H
( )
... condition


Example
114
( ) ( ) ( )
( )
( )
   
( )
1
0 0 0
1
1
1 2 1 2
0 2 2
5
5.
10
5 5 3.75
6. 0.25
10 10 7.50
6 2 2 4 37.5 37.5
37.5 37.5 37.5 2 ...
3. ...
x x x x
convergence condition not
satisfied
continue

− −
 
 
= − =  
  −
 
−
     
= + =
     
−
     
= + + =
= + = 
d H c
X
c
c

Example
115
( )
( )
( )
4 2 2 2
1 1 2 2 1 1
0
10 20 10 2 5
1,3 , 0.0001
f x x x x x x

= − + + − +
= − =
X
X

Adv/disadv
• Comparatively more time consuming calculations
such as Hessian (second order derivatives), may
not be possible for some problems.
• Positive definiteness of the Hessian is the must to
confirm the search direction is the desirable one.
• Very quick to converge for convex functions
• Quadratic rate of convergence and hence
converges very fast.
• In case of the large sized problems, inversions are
too difficult.
116

Quasi-Newton Method
• Use of first-order information to generate the
Hessian and/or its inverse, i.e. incorporating
the advantages of both ‘Steepest Descent’ and
‘Newton Method’. ….Hybrid???
117

Quasi-Newton Method:
Davidon-Fletcher-Powell (DFP) Method
Approximates the ‘Inverse of Hessian’ using
only first order derivatives
118

Iteration 1 (similar as Steepest Descent)
point , i.e.
solution.
119
( ) ( )
( )
( )
( ) ( )
( ) ( )
( )
1 2
...
k k k
k k
n
f f f
f
x x x
 
  
 
=  =  
  
 
 
X X X
c X
( )
k
X
0.0001
 
( )
f X
( )
k
X
( )
k


c
( )
k
X ( )
f X

4. Choose a symmetric positive definite
inverse of Hessian . Generally, .
5. Calculate the search direction and
check whether it is ‘desirable’ using following
condition
6. Calculate (pre-specify) the step size .
.
120

( ) ( ) ( )
1
k k k

+
= +
X X d
( )
( )
1
k
f +
X
( )
1
k+
X
n n

( )
k
A ( )
k
=
A I
( ) ( ) ( ) ( ) ( )
( )
0 0
T T
k k k k k
   
    
   
c d c A c
( ) ( ) ( )
k k k
= −
d A c

Iteration 2 and further
8. Update the inverse of the Hessian
where
and…
121
( ) ( ) ( ) ( )
1
k k k k
+
= + +
A A B C
( )
( ) ( )
( ) ( )
T
k k
k
k k
 
 
=

s s
B
s y
( )
( ) ( )
( ) ( )
T
k k
k
k k
 
−  
=

z z
C
y z

9. Go to step 2
122
( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( )
( )
( )
( ) ( )
( ) ( )
( )
1
1 1 1
1
1 2
; ; .
...
k k k k k k k k k
k k k
k
n
f f f
x x x
 +
+ + +
+
= = − =
 
  
 
=  
  
 
 
s d y c c z A y
X X X
c

Quasi-Newton Method: Broyden-
Fletcher-Goldfarb-Shanno (BFGS)
Method
Approximates the ‘Hessian’ using only first
order derivatives. Also referred to as ‘Direct
Hessian Updating’ method.
123

Broyden-Fletcher-Goldfarb-Shanno
(BFGS) Method
Iteration 1 (same as Steepest Descent)
point , i.e.
solution.
124
( ) ( )
( )
( )
( ) ( )
( ) ( )
( )
1 2
...
k k k
k k
n
f f f
f
x x x
 
  
 
=  =  
  
 
 
X X X
c X
( )
k
X
0.0001
 
( )
f X
( )
k
X
( )
k


c
( )
k
X ( )
f X

(BFGS) Method
4. Choose a symmetric positive definite
Hessian . Generally, .
5. Calculate the search direction by solving the
system of linear equations to
find the search direction .
6. Calculate (pre-specify) the step size .
.
125

( ) ( ) ( )
1
k k k

+
= +
X X d
( )
( )
1
k
f +
X
( )
1
k+
X
n n

( )
k
H ( )
k
=
H I
( ) ( ) ( )
k k k
= −
H d c
( )
k
d

(BFGS) Method
Iteration 2 and further
8. Update the inverse of the Hessian
where
and…
126
( ) ( ) ( ) ( )
1
k k k k
+
= + +
H H D E
( )
( ) ( )
( ) ( )
T
k k
k
k k
 
 
=

y y
D
y s
( )
( ) ( )
( ) ( )
T
k k
k
k k
 
 
=

c c
E
c d

(BFGS) Method
9. Go to step 2
127
( ) ( ) ( ) ( ) ( ) ( )
( )
( )
( ) ( )
( ) ( )
( )
1
1 1 1
1
1 2
;
...
k k k k k k
k k k
k
n
f f f
x x x
 +
+ + +
+
= = −
 
  
 
=  
  
 
 
s d y c c
X X X
c

Multi-Criteria Decision Making
128

Weighted Scoring Model/Weighted
Sum Model
• Prioritizing Requirements
• Rank
• Systematic Process
129
It is very important to state here that it is
applicable only when all the data are
expressed in exactly the same unit. If this
is not the case, then the final result is
equivalent to "adding apples and oranges."

• Criteria for Weighing
– Value
– Risk
– Difficulty
– Success
– Compliance
– Relationships
– Stakeholder
– Urgency
130
Assign % or
Score to each
criterion

Example 1
Requirement Score
Criteria Weight A B C
X 50% 70 45 40
Y 30% 40 85 30
Z 20% 40 80 50
Weighted Scores 100% 55 64 39

Example 2
Requirement Score
Criteria Weight A B C D E
Value 20% 80 45 40 15 35
Risk 20% 60 85 30 20 75
Difficulty 15% 55 80 50 15 25
Success 10% 30 60 55 65 30
Compliance 5% 35 50 60 50 50
Relationships 5% 80 70 50 85 80
Stakeholder 15% 25 50 45 60 60
Urgency 10% 60 25 40 65 80
Weighted Score 100% 54.8 60.0 43.3 38.0 52.3

Analytic Hierarchy Process (AHP)
• Information is decomposed into a hierarchy of
alternatives and criteria
• Information is then synthesized to determine
relative ranking of alternatives
• Both qualitative and quantitative information
can be compared using informed judgements
to derive weights and priorities

Example: Car Selection
• Objective
– Selecting a car
• Criteria
– Style, Reliability, Fuel-economy
• Alternatives
– Civic Coupe, Saturn Coupe, Ford Escort, Mazda
Miata

Hierarchical tree
Style Reliability Fuel Economy
Selecting
a New Car
- Civic
- Saturn
- Escort
- Miata
- Civic
- Saturn
- Escort
- Miata
- Civic
- Saturn
- Escort
- Miata
Objective
Criteria
Alternatives

Mat
Cost
Mfg
Cost
Reparability Durability Reliability Time to
Produce
Mat Cost
Mfg Cost
Reparability
Durability
Reliability
Time to
Produce

Mat Cost Mfg Cost Reparability Durability Reliability Time to
Produce
Mat Cost 1 0.33 0.2 0.11 0.14 3
Mfg Cost 3 1 0.33 0.14 0.33 3
Reparability 5 3 1 0.2 0.2 3
Durability 9 7 5 1 3 7
Reliability 7 3 5 0.33 1 9
Time to
Produce
0.33 0.33 0.33 0.14 0.11 1
Sum 25.33 14.66 11.86 1.92 4.78 26
139

140
Produce
Mat Cost 1 0.33 0.2 0.11 0.14 3
Mfg Cost 3 1 0.33 0.14 0.33 3
Reparability 5 3 1 0.2 0.2 3
Durability 9 7 5 1 3 7
Reliability 7 3 5 0.33 1 9
Time to
Produce
0.33 0.33 0.33 0.14 0.11 1
Sum 25.33 14.66 11.86 1.92 4.78 26
Produce
Mat Cost 0.0394789 0.02251 0.016863406 0.057292 0.0292887 0.115385
Mfg Cost 0.1184366 0.06821 0.027824621 0.072917 0.0690377 0.115385
Reparability 0.1973944 0.20464 0.084317032 0.104167 0.041841 0.115385
Durability 0.3553099 0.47749 0.42158516 0.520833 0.6276151 0.269231
Reliability 0.2763522 0.20464 0.42158516 0.171875 0.209205 0.346154
Time to
Produce
0.013028 0.02251 0.027824621 0.072917 0.0230126 0.038462
Sum 1 1 1 1 1 1

Produce
Criterion
Weights
Mat Cost 0.0394789 0.02251 0.016863406 0.057292 0.0292887 0.115385 0.04680292
Mfg Cost 0.1184366 0.06821 0.027824621 0.072917 0.0690377 0.115385 0.0786355
Reparability 0.1973944 0.20464 0.084317032 0.104167 0.041841 0.115385 0.1246237
Durability 0.3553099 0.47749 0.42158516 0.520833 0.6276151 0.269231 0.445344
Reliability 0.2763522 0.20464 0.42158516 0.171875 0.209205 0.346154 0.27163494
Time to
Produce
0.013028 0.02251 0.027824621 0.072917 0.0230126 0.038462 0.03295894
Sum 1 1 1 1 1 1
141

No Logical Errors A>B>C so A>C
{Ws} Consis CI Random
Index (RI)
0.286 6.093
0.515 6.526 0.1219 1.25
0.839 6.742
3.09 6.95
1.908 7.022 CI/RI= 0.09752 So Consistent
0.21 6.324
Average (λ) 6.6095
Number of
Criteria (n)
6
143

Design Alternatives
145
Material Cost [C]
Plate Welds Plate Rivets Casting
Plate Welds 1 1 0.33333
Plate Rivets 1 1 0.33333
Casting 3 3 1
Sum 5 5 1.66666
Normalized [C] Design Alternatives Priorities {P}
Plate Welds Plate Rivets Casting Average of Rows
Plate Welds 0.2 0.2 0.2 0.2
Plate Rivets 0.2 0.2 0.2 0.2
Casting 0.6 0.6 0.6 0.6
Ws = [C]*{P} Consis = Ws/{P} Number of Criteria (n)= 3
0.6 3 RI = 0.58
0.6 3 CI= 0
1.8 3 CR= 0
Average (λ) = 3

146
Plate Welds Plate Rivets Casting {W}
Mat Cost 0.2 0.2 0.6 0.046802917
Mfg Cost 0.26 0.633 0.106 0.078635503
Reparability 0.292 0.615 0.093 0.124623697
Durability 0.429 0.429 0.143 0.445344
Reliability 0.26 0.633 0.105 0.271634942
Time to Produce 0.269 0.633 0.106 0.03295894
Plate Welds 0.2 0.26 0.292 0.429 0.26 0.269 0.046803
Plate Rivets 0.2 0.633 0.615 0.429 0.633 0.633 0.078636
Casting 0.6 0.106 0.093 0.143 0.105 0.106 0.124624
0.445344
0.271635
0.032959
Plate Welds 0.33665
Plate Rivets 0.519637
Casting 0.143799

Introduction of Lagrange Multipliers
151

Random Search: Random Jumping
Method
where
152
; 1,2,...,
l i u
x x x i n
  =
( ) ( )
1 2
, ,..., n
f f x x x
=
X
( )
( )
( )
( )
1, 1 1, 1,
1
2 2, 2 2, 2,
1
1 , , ,
1
l u l
l u l
n n l n n u n l
x r x x
x
x x r x x
f
x x r x x
 
+ −
 
 
  + −
 
 
= 
   
   
   
+ −
   
X
( )
( ) ( )
1 2
1 2
, ,..., 0,1
. . , ,..., 0,1
n
n
r r r are random numbers from
i e r r r 

Random Search: Random Jumping
Method
choose very
large value of m
153
( )
( )
( )
( )
( )
( )
( )
1
1, 1 1, 1,
1
2 2, 2 2, 2,
2
2 , , ,
2
3
l u l
l u l
n n l n n u n l
m
f
x r x x
x
x x r x x
f
x Choose the best out them
x r x x
f
f



  
+ −
 
  
  + −
 
  
= 
    
    

   
+ −
  
 







 

X
X
X
X

Random Search: Random Walk
Method
2. Generate a set of random numbers
and formulate the search direction vector
154
( )
k
X
0.0001
 
( ) ( )
1 2
, ,..., 1,1
n
r r r  −
n
( )
1 1
2 2
2 2 2
1 2
1 1
...
k
n
n n
r r
r r
R r r r
r r
   
   
   
= =
   
+ + +
   
   
   
d

Method
3.
155
( )
( )
( ) ( )
( ) ( )
2 2 2
1 2
1 2
1 2
... . . 1 .
1 ...
, ,..., 1,1
...
, ,..., 1,1
k
n
k
n
n
if r r r i e R accept and continue
elseif R discard and
find the new set of random numbers r r r
and continue till
found a new set of random numbers r r r
satisfying the condition R
+ + + 

  −
  −
d
d
1.


Method
4. … once found, update
and compute
5. Go to step 2
…continue till convergence, i.e.
BUT this may happen ???
Go back to step 2.
156
( ) ( ) ( )
1
k k k

+
= +
X X d
( )
( )
1
k
f +
X
( )
( ) ( )
( )
1
k k
f f
+

X X
( )
( ) ( )
( )
1
k k
f f 
+
− 
X X

Adv.
• This method works for any kind of function
(discontinuous, non-differentiable, etc.)
• Useful when function has several local minima
• Useful especially, to locate the region in the
neighborhood of global minimum.
157

Random Search: Constrained
158

Random Search: Constrained
• Penalty Function Method:
Will be discussed very soon
159

Grid Search Method
• Set up a grid in the design space
choose points in the
range inclusive.
Requires a large
number of grid
points to be evaluated:
no. of grid points =
= no. of variables
160
1
x
2
x
1,u
x
1,l
x
2,l
x
2,u
x
n
m
( )
, ,
,
i l i u
x x
n
n

Simplex Method: Example 1
161
1 2 3
1 2 3
1 2 3
1 2 3
9 2 3
2 3 5
2 2 2
, , 0
Minimize f x x x
subject to
x x x
x x x
x x x
= + +
+ −  −
− +  −


Solution of Constrained Problems
using Unconstrained Optimization
Methods
• Basic Idea: Form a composite function using
the cost and constraint function
parameterized by a penalty parameter.
• Penalty parameter applies larger penalty for
larger constraint violation and smaller penalty
for smaller violation.
• Penalty parameter is adaptive specific to the
magnitude of the constraint violation
162

Solution of Constrained Problems
using Unconstrained Optimization
Methods
• Referred to as Penalty Function method,
163
( )
( )
( )
( ) ( )
( ) ( ) ( ) ( )
( ) ( )
( )
2 2
1 1
0 1,...,
0 1,...,
, ,
max 0,
i
j
p m
i j
i j
j j
Minimize f
h i p
g j m
r f r h g
where g g
 +
= =
+
= =
 =
 
 
 
 
= + +
 
   
 
 
=
 
X
X
X
h X g X X X X
X X

Adv and Disadv
• Equality and inequality constraints can be
easily accommodated
• Can be started from any starting point (no
need to have an educated guess)
• The design space needs to be continuous
• The rate of convergence essentially depends
on the rate of change of penalty parameter.
164

Barrier Function Method
• The initial solution needs to be feasible
• Barriers are constructed around the feasible
region
165
( )
( )
( )
( ) ( )
( )
( ) ( )
( )
1
1
0 1,...,
1 1
,
1
, log
j
m
j j
m
j
j
Minimize f
g j m
r barrier function method
r g
r g log barrier function method
r


=
=
 =
−
=
= −


X
X
g X
X
g X X

Barrier Function Method
• The barriers do not allow the solution to go
out of the feasibility region and maintains the
feasibility by imposing the penalty.
• For both the methods:
166
( ) ( )
*
r f f
→   →
X X

Adv and Disadv of Barrier Function
Method
• Applicable to inequality constraints only
• Starting design point needs to be feasible
however, this makes the final solution to be
acceptable.
167

Adv and Disadv of Barrier Function
Method
• These method tend to perturb itself at the
boundary of the feasible region where true
optimum may be located because
may impose high penalty.
• There are chances of Hessian becoming ill-
conditioned when (may become
singular).
168
r very high
→
r very high
→

Multiplier Method, Augmented
Lagrangian Method
• Some of the disadvantages are removed such
as and so the chances of Hessian
becoming ill-conditioned and may become
singular.
169
r very high
→

Multiplier Method, Augmented
Lagrangian Method
170
( )
( )
( )
( ) ( )
( ) ( ) ( )
2
2
' '
1 1
0 1,...,
0 1,...,
1 1
, , ,
2 2
i
j
p m
i i i j j i
i j
Minimize f
h i p
g j m
r h r g
  
+
= =
= =
 =
 
 
 
= + + +
 
 
 
 
 
 
X
X
X
h X g X r θ

Linearization of Constrained Problem
171
( )
( )
( )
( ) ( )
( ) ( )
( ) ( )
( ) ( )
( ) ( )
( ) ( )
( ) ( )
( ) ( )
( ) ( )
( ) ( )
( ) ( )
( ) ( )
0 1,...,
0 1,...,
' :
0 1,...,
0 1,...,
j
j
k k k k k
T
k k k k k
T
j j j
k k k k k
T
j j j
Minimize f
h j p
g j m
Taylor s Expansion as the basis of iterative optimization
Minimize f f f
Subject to
h h h j p
g g g j m
= =
 =
+   +  
+   +   = =
+   +    =
X
X
X
X X X X X
X X X X X
X X X X X

• Generalized Representation ??
172
( )
( )
( )
( )
( )
( )
( )
( ) ( )
( ) ( )
( )
( )
, ,
k
k
k
j j
k
j j
k k k
j j
i ij ij
i i i
k
i i
f f
e h
b g
f h g
c n a
x x x
d x
=
= −
= −
  
= = =
  
= 
X
X
X
X X X

• Generalized Representation
173
1
1
1
1,...,
1,...,
n
T
i i
i
n
T
ij i j
i
n
T
ij i j
i
Minimize f c d f
Subject to
n d e j p
a d b j m
=
=
=
=  =
= =  =
 =  



c d
N d e
A d b

Illustrative Example
174
( )
( )
( )
( )
( )
( )
2 2
1 2 1 2
2 2
1 1 2
2 1
3 2
0
3
1 1
1.0 0
6 6
0
0
1,1
Minimize f x x x x
Subject to
g x x
g x
g x
x
= + −
= + − 
= 
= 
=
X
X
X
X

• Gradients of Cost Function and constraint
Functions are
176
( )
( )
( )
( )
( )
( )
( ) ( )
( )
( )
( )
1 2 2 1
1 1 2
2
3
0 0
0
0
1
2 3 , 2 3
2 2
,
6 6
1,0
0, 1
1, 1
1,1 1 1
,
3 3
f x x x x
g x x
g
g
f
g

 = − −

  
 =   
  

 = −


 = − 

 = = − −

 =  
 
 = 
 
  
X c
X
X

177
( )
( )
( )
( )
( )
( ) ( )
( ) ( )
( )
( )
( ) ( )
( )
( )
( )
0
0
0 0 0
1 1 2 2 3 3
0 0
0
1
1,1
1.0
2
, 1, 1
3
1, 1
1 1
,
3 3
f
b g b g b g
f
g
 = 
= −
= = = = = =
 = = − −
 
 =  
 
X
X
X X X
X c
X

178
( )
( )
( )
( )
( )
( )
0
1 1
0
2 2
0
3 3
2
2
1 3
1 0 3
3
, 1 1
1
0 1 1
1
3
b g
b g
b g
 
  = =
 
   
−  
   
 
= = = = =
   
 
   
−  
  = =
 
 
 
   
X
A b X
X
( )
( )
0
1
1 1
,
3 3
g
 
 =  
 
X
( )
2 1,0
g
 = − ( )
3 0, 1
g
 = −

179
  1
1 2
1
1
2
1 1
1,...,
2
1
1 0 3
3
1
1
0 1 1
3
n
T
i i
i
n
T
ij i j
i
d
Minimize f c d f
d
Subject to
a d b j m
d
d
=
=

 
=  = = − −  
 
 =  
 
   
−
   
 

   
 
 
   
−
   
 
 


c d
A d b

• Expanded:
• This is the linearized problem in terms of:
180
1 2
1 2
1
2
1 1 2
,
3 3 3
1 ,
1
Minimize f d d
Subject to
d d
d
d
= − −
+ 
− 
− 
1 2
d and d

181
• Linearization in terms of .
1 2
x and x

Sequential Linear Programming
Algorithm
• Linear format
Must be +ve
182
1
1
1
1,...,
1,...,
n
T
i i
i
n
T
ij i j
i
n
T
ij i j
i
Minimize f c d f
Subject to
n d e j p
a d b j m
=
=
=
=  =
= =  =
 =  



c d
N d e
A d b

Algorithm
• As the linearized problem may not have bounded
solution, the limits on the design needs to be
imposed.
The design is
still linear abd
can be solved
using LP
Methods
183
( ) ( )
, 1,...,
k k
il i iu
d i n
−    =

Algorithm
• Selection of the proper limits
requires some
problem information/knowledge
• Should choose based on the trial and error or
should be made adaptive. This may help
avoiding the entry in the infeasible region.
184
( ) ( )
, , 1,...,
k k
il iu
and i i n
   =

Approximation Algorithms
• Heuristics
• Metaheuristics
186

189
Nature Inspired Algorithms
Bio-Inspired Algo.
Evolutionary Algo.
Genetic Algorithm
Evolution Strategies
Genetic Programming
Evolutionary Programming
Differential Evolution
Cultural / Social
Algorithm
Artificial Immune
System
Bacteria Foraging
& many others
Swarm Intelligence
Algo.
Ant Colony
Cat Swarm Opt.
Cuckoo Search
Firefly Algo
Bat algorithm
Physical, Chemical
Systems
Based Algo.
Simulated
Annealing
Harmony Search

190
Cultural / Social Algorithm (SA)
SA based on Socio-
Political Ideologies
Ideology Algorithm
Election Algorithm
Election Campaign
Algorithm
SA based on Sports
Competitive Behavior
Soccer League
Competition Algorithm
League Championship
Algorithm
SA based on Social and
Cultural Interaction
Cohort Intelligence
Teaching Learning
Optimization
Social Group
Optimization
Social Learning
Optimization
Cultural Evolution
Algorithm
Social Emotional
Optimization
Socio Evolution &
Learning Optimization
SA based on
Colonization
Society and civilization
Optimization Algorithm
Imperialist Competitive
Algorithm
Anarchic Society
Optimization

Metaheuristics
• Nature Inspired Methods
– Bio-inspired Methods
– Swarm Based Methods
191

Genetic Algorithm
• John Holland introduced GA
in 1960 inspired from
Darwin’s Theory of Evolution
and Survival of Fittest.
• GA became Popular when
first book published:
‘Adaptation in Natural and
Artificial Systems’ by John
Holland in 1975 and
applications came up in
1980.
197

Genetic Algorithm
• Exploits the evolution through
– Inheritance
– Selection
– crossover
– Mutation
198

GA Operators: Crossover &Mutation
200

Termination
• The algorithm terminates if the population has
converged, i.e. population does not produce
offspring which are significantly different from
the previous generation).
201

Illustrative Example 2
• The OneMax or MaxOne Problem (or
BitCounting) is a simple problem consisting in
maximizing the number of ones of a bitstring.
203

Optimization Techniques.pdf

More Related Content

What's hot (20)

Similar to Optimization Techniques.pdf (20)

Recently uploaded (20)

Optimization Techniques.pdf