Introduction to optimization technique

Introduction to Optimization
& it's Techniques
Submitted by
Kamini Singh (Ph.d Scholar)
IME Department, IIT Kanpur

Contents
• What is an optimization?
• Components of an optimization problem
• Classification of optimization problem
• Optimization Taxonomy
• Algorithm by problem type
• Advanced Optimization Technique
• Software Availability
• References
3/22/2018
KaminiSingh,IMEDept.IITKanpur
2

What is an optimization ?
• The action of making the best or most effective use of given
situation - (Google dictionary)
• It is a technique of squeezing best performance out of
provided current state of model
3/22/2018
3

Components of an optimization
problem
• Objective function
An objective function express performance of a system, need
to be minimized or maximized
• Variable (Design or Decision Variable)
A set of unknowns, define the objective function and
constraints, can be continuous, discrete or boolean
• Constraints
They are conditions, allows the unknowns to take on certain
values but exclude others to render the design to be feasible
3/22/2018
4

An example
• Objective function
Min/Max ƒ(𝑥𝑖, … . , 𝑥 𝑛)
• Constraints
𝑔𝑖 𝑥𝑖, … . , 𝑥 𝑛 = 0 ∀ 𝑖 𝜖 1, … , 𝑝
𝚽𝑗 𝑥𝑖, … . , 𝑥 𝑛 ≤/ ≥ 0 ∀ 𝑗 𝜖 1, … , 𝑚
• Variable
𝑥 is an n- dimensional vector called design variable; 𝑥 =
𝑥1
⋮
𝑥 𝑛
3/22/2018
5
Equality Constraints
Inequality Constraints

Types of optimization problem
• Based on existence of constrains
• Based on the nature of the equations involved
• Based on the permissible values of the decision variables
• Based on the number of objective functions
3/22/201
8
6

Based on existence of constrains (1/4)
• Unconstrained Optimization Problem (No Constraints)
Objective function
• Constrained Optimization Problem (One or more than one constrained)
Objective function
Subject to:
𝑔𝑖 𝑥𝑖, … . , 𝑥 𝑛 = 0 ∀ 𝑖 𝜖 1, … , 𝑝
𝚽𝑗 𝑥𝑖, … . , 𝑥 𝑛 ≤/ ≥ 0 ∀ 𝑗 𝜖 1, … , 𝑚
3/22/2018
7
Equality Constraints
Inequality Constraints

Based on the nature of the equations
involved (2/4)
• Linear Programming Problem (LPP)
Linear objective function with linear constraints
• Quadratic Problem 𝑋 =
𝑥
𝑦 and 𝐴 =
𝑎 𝑏 2
𝑏 2 𝑐
; 𝑓 𝑋 = 𝑋 𝑇 𝐴 𝑋
• Non-Linear Programming Problem (NLP)
3/22/2018
8
Maximize 𝐶 𝑇
𝑥
Subject to 𝐴𝑥 ≤ 𝐵
𝑥 ≥ 0
Standard
form
Minimize 𝑓 𝑥, 𝑦 = 𝑎𝑥2
+ 𝑏𝑥𝑦 + 𝑐𝑦2
Min/Max 𝑓 𝑥 = 𝑥1 𝑥2 𝑥3
Subject to 2𝑥1 + 4𝑥2 𝑥3 ≤ 10
𝑥1 + 𝑥2 + 𝑥3 ≤ 7
8𝑥1 + 𝑥2 + 5𝑥3 < 4
𝑥𝑖 ≥ 0 ∀ 𝑖 = 1,2, 3
(with constraints it is become Quadratically
Constrained Quadratic Programming problem )

Based on the permissible values of the
decision variables (3/4)
• Integer Programming
Problem (an example of LPP)
• Decision variables (𝑥1, 𝑥2) are
allowed to take only integral
value
3/22/2018
9
• Real-Valued Programming
Problem
• Decision variable allowed to
take any real value in its
domain
• 𝑥𝑖 ≥ 0, ∀ 𝑖 = 1, … , 𝑛
2𝑥1 + 3𝑥2 ≤ 18, 𝐿1
𝑥1 + 3𝑥2 ≤ 9, 𝐿2
𝑥2
𝑥1 The dot represents
the Integer point set
on 2-D coordinate
Optimal Value
𝐿1
𝐿2
2𝑥1 + 3𝑥2 ≤ 18, 𝐿1
𝑥1 + 3𝑥2 ≤ 9, 𝐿2
𝑥2
𝑥1
Optimal Value
𝐿1
𝐿2

Based on the number of objective
functions (4/4)
• Single objective
function problem
• Problem in which there is only
single objective function for 𝑥𝑖,
need to be minimized/maximized
Objective function
Min/Max
ƒ(𝑥𝑖, … . , 𝑥 𝑛)
Subject to
𝑔𝑖 𝑥𝑖, … . , 𝑥 𝑛 = 0
∀ 𝑖 𝜖 1, … , 𝑝
3/22/2018 10
• Multi-objective function
problem
• More than one objective function
for 𝑥𝑖 , need to be
minimized/maximized
Objective function
Min/Max
ƒ(𝑥1) 𝑓 𝑥2 … . 𝑓(𝑥 𝑛)
Subject to
𝑔𝑗 𝑥 = 0
∀ 𝑗 𝜖 1, … , 𝑝

Optimization Taxonomy (1/2)
3/22/2018
11NOTE: Unconstrained branch is opened in next slide

Optimization Taxonomy (2/2)
3/22/2018
12NOTE: Constrained branch is opened in previous slide

Algorithm by problem type (Traditional)
3/22/2018
13
Unconstrained Optimization
• Line search Methods
• Trust-Region Methods
• Tensor Methods
• Homotopy Methods
• Truncated Newton Methods
• Difference Approximations
• Quasi-Newton Methods
• Nonlinear Conjugate Gradient Method
• Nonlinear Simplex Method
• The Gauss Newton Method
• The Levenberg-Marquardt Method
• Hybrid Methods
• Large-Scale Methods
Nonlinear
Equations
Nonlinear
Least
Square
Constrained Optimization
• Newton Methods
• Gradient Projection Methods
• Simplex Method
• Interior Point Methods
• Algorithms for Quadratic
Programming
• Augmented Lagrangian Methods
• Reduced Gradient Methods
• Sequential Quadratic Programming
• Feasible Sequential Quadratic
Programming
• Central Cutting Plane Methods
• Discretization Methods
• KKT Reduction Methods
• SQP Reduction Methods
Bound Constrained
Optimization
Quadratic
Programming
Linear
Programming
Quadratic
Programming
Semi-infinite
Programming

Advanced Optimization Techniques
(Non-Traditional)
• Evolutionary Algorithm (Genetic Algorithm)
• Trajectory Based Method (Simulated Annealing)
• Hill Climbing
• Particle Swarm Intelligence
• Pattern Search
• Ant Colony Optimization
• Tabu Search
• Fuzzy Optimization
3/22/2018
14

Genetic Algorithm (1/3)
• Introduced by John Holland in
1970s, which is continuously
evolved by his students and
followers over the years
• A strong tool with adaptive
heuristic search algorithm based on
evolutionary ideas
• Facilitate a more robust Artificial
Intelligence
• Terminology used in GA
Chromosome (String)
Genes (Bits)
Locus
Allele (Value)
3/22/2018
15
1 0 1 1 0
0
0
Chromosome
(String)
Gene (Bit)
Allele
Locus

• The Genetic operator
Reproduction
Crossover
Mutation
3/22/2018
16
Crossover
Types of Mutation
Reproduction

• Population size (n)
 In GA, it is integration of
predecessors with their
reproduction, crossover and
mutation
• Fitness Value
It is the same as the
objective function
(sometime for minimization
problem number of conversion is
done to get optimum way using
exponential function)
3/22/2018
17
Start
Generate Initial Population
Evaluate Fitness function
Stop
Re-generate
process
Selection process
Meets
Optimization
Criteria
Yes
No

Simulated Annealing(1/2)
• Simulated annealing (SA)
is a probabilistic technique
for approximating
the global optimum of a
given function
• Technique is inspired from
the annealing process of
metallurgy as name goes
• It search for the global
maxima in the
neighborhood solution
3/22/2018
18
Thermodynamic
Simulation
Combinatorial
Optimization
System State Feasible Solution
Energy Transportation Cost
and Elapsed Time
Change of State Neighboring
Solutions
Temperature Control Parameter
Frozen State Heuristic Solution
Comparison between physical and
simulated annealing

Simulated Annealing (2/2)
• It follows the exponential
Boltzmann probability (K
is a Boltzmann constant in
equation [𝑝 = 𝑒(−∆ 𝑇∗𝐾)] )
• ∆ compares the previous
and current objective
function
3/22/2018
19
Calculate 𝑓 𝑥 for trial solution and set
∆ = 𝑓 𝑥 𝑘+1 − 𝑓(𝑥 𝑘)
Accept the Trial
Solution
Start
Select initial Temperature 𝑇0
Generate a trial solution in the neighborhood of present solution
Stop
K= k+1
Decrease the temperature by, 𝑇𝑘+1 = 𝑝 ∗ 𝑇𝑘
∆< 0?
Yes No
Accept the trial solution
with probability, p
(𝑒(−∆ 𝑇∗𝐾)
) > 𝑟
Freezing
point ?
No
No
Yes

Hill Climbing Optimization (1/2)
• Hill climbing is a mathematical
optimization technique which
belongs to the family of local
search with iterative algorithm
• Starts with an arbitrary solution to
a problem and attempts to find a
better solution by incrementally
changing a single element of the
solution
• Algorithm Components
Local Maxima (not highest)
Plateau (Random walk)
Ridges (flat like a plateau, but with
drop-offs to the sides)
3/22/2018
20
Local
Maxima Global
Maxima
Plateau
𝑓(𝑥)
State Space (𝑥)
3-D View
Current
state
Shoulder/Plateau
Local
Maxima
𝑓(𝑥)
State Space (𝑥)
Global
Maxima
2-D View

Hill Climbing Optimization (2/2)
3/22/2018
21
Local Maxima
Global
Maxima
Plateau
• Not complete since the search will
terminate at local minima, plateaus, and
ridges.
Start
Look one step ahead
Stop
Re-generate
process
k= k+1
𝑓(𝑥 𝑘+1) < 𝑓 𝑥 𝑘
∀ 𝑥𝑖 ≤ 𝑥 𝑘
Yes
No

Particle Swarm Intelligence (1/2)
3/22/2018
22
• Particle swarm optimization
(PSO) is a computational
method to optimize a
problem
• It works iteratively trying to
improve a candidate
solution with regard to a
given measure of quality
• It studies the position and
velocities of particle at time
(t)
Random Fly
Personal
best (pbest)
Ordered Fly
Personal best
become global
best (gbest)
Fish following
global best
(gbest)

Particle Swarm Intelligence (2/2)
• pBest: personal best solution of
individual bird in flock
• gBest: Global best of the bird
flock
3/22/2018
23
Assign current
fitness as new
pBest Solution
Start
Generate Initial Population
Evaluate Fitness function for each particle, ∆ = 𝑓 𝑥 𝑘+1 −
𝑓(𝑥 𝑘)
Use each particle’s velocity value to update its data
values
Stop
Calculate Velocity for each particle
𝑖𝑓 ∆ is
better than
pBest ?
Yes No
Keep previous
best
Target
reached?
No
No Yes
Assign best particle’s pBest value to gBest

Pattern Search (1/2)
• This methodology involves setting up of grids in the decision space
• Evaluates the values of the objective function at each grid point
• To get the optimum point it increment and decrement the grid space
• The best value of the objective function is corresponding to the optimum
point is considered to be an optimum solution
• Stages involved in it are
Pattern (generation as per GPS (Using ‘N’ or ‘N+1’ Law)
Meshes (for each point search for a new algorithm)
Polling (Polls the point obtained after meshing to meet objective function)
Increment and Decrement (as per polling it expands and contracts the pattern to get the
optimal solution)
3/22/2018
24

Pattern Search (2/2)
3/22/2018
25
Poll = Minimum objective in the pattern at point (k+1) −Current point value (k)
Start
Generate a pattern of points (both direction of coordinates), specify mesh
size and center this pattern on the current point
Evaluate Fitness function at every point
Stop
k= k+1
Half the Mesh size
Poll < 0
Yes
Nok= k+1, Double
the Mesh Size
Mesh size <
Threshold
Yes
No
• Pattern search finds a
local minimum of an
objective function by
using polling
• Example

Ant Colony Optimization (1/2)
• Ant Colony Optimization is
based on the real life example of
ant
• It belongs to the iterative and
sub-group of evolutionary family
as same goes to Particle Swarm
Intelligence
• Ants follows the pheromone
chemical leave behind by their
successor to find the shortest path
3/22/2018
26
A real image of Ant alignment
towards food

Ant Colony Optimization (2/2)
3/22/2018
27
Every ant schedule its next
operation
Update probability values each edge
to feasible node
Pheromone Evaporation
Start
Select Initial Parameter
All ants schedule their own first operation
Stop, schedule the
operation
Ant reached
food node?
Yes
Analysis of best
solution found and
update pheromones
according to criteria
Ants
accomplished all
needed trip?
No
No
Yes
Each ant updates its list of feasible operation
• Probability 𝐨𝐟 𝒌 𝒕𝒉ant move from 𝒙 to 𝒚
𝑝 𝑥𝑦
𝑘
= (𝜏 𝑥𝑦
𝛼
)(𝜃 𝑥𝑦
𝛽
) 𝑧𝜖𝑎𝑙𝑙𝑜𝑤𝑒𝑑 𝑥𝑦
(𝜏 𝑥𝑦
𝛼
)(𝜃 𝑥𝑦
𝛽
)
𝜏 𝑥𝑦: amount of pheromone deposited from transition
x to y, 𝛼 ≥ 0 (𝑐𝑜𝑛𝑡𝑟𝑜𝑙 𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑒𝑟 𝑓𝑜𝑟 𝛼)
𝜃 𝑥𝑦: prior knowledge of state 𝑥𝑦 (1
𝑑 𝑥𝑦
, d is a
distance), 𝛽 ≥ 1 (𝑐𝑜𝑛𝑡𝑟𝑜𝑙 𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑒𝑟 𝑓𝑜𝑟 𝜃)
• Pheromone Update
Trials updated by 𝜏 𝑥𝑦 ← (1 − 𝜌)𝜏 𝑥𝑦 + 𝑘 ∆𝜏 𝑥𝑦
𝑘
)
𝜌: evaporation coefficient
∆𝜏 𝑥𝑦
𝑘
=
𝑄
𝐿 𝑘
0
𝑖𝑓 𝑎𝑛𝑡 𝑘 𝑢𝑠𝑒𝑠 𝑐𝑢𝑟𝑣𝑒 𝑥𝑦 𝑖𝑛 𝑖𝑡𝑠 𝑡𝑜𝑢𝑟
𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
𝐿 𝑘 is the cost of the kth ant’s tour and Q is a
constant

Tabu Search (1/2)
• Tabu (prevent cycling) search introduced by Fred W. Glover in
1986
• It is a metaheuristic based local search methods used
• At each step worsening moves can be accepted, if solution get
stuck to a strict local minim
• A forbidden (Tabu) is introduced to discourage the entrapment to
previously-visited (strict local minima) solutions
3/22/2018
28

Tabu Search (2/2)
• X: set of feasible solution
• 𝑥: Current solution, 𝑥 ∈ 𝑋
• 𝑥′′
: Best solution reached
• 𝑥′: Best solution among sample of trialed solution
• 𝑓(𝑥): evaluation function of 𝑥
• N(𝑥): Neighborhood set, 𝑥 ∈ 𝑋 (Trial solution)
• S(𝑥): Neighborhood sample of 𝑥, S(𝑥) ∈ N(𝑥)
• SS(𝑥): Sorted sample in ascending order as per 𝑓(𝑥)
3/22/2018
29
Yes
Every ant schedule its next
operation
Calculate the objective function to each neighbor in the
set, and find the best among them
Find a set of neighbors to the current solution
Start
Select Initial Parameter (TL & AV)
Find randomly an initial solution
Stop
Is the best neighbor
TABU ?
Yes
Is AV
satisfied?
No
No
Yes
Set the current and best solution equal to the initial solution
Is stopping
criteria satisfied
No
• Initialization Parameter
 Set TL (Tabu list restriction) and AV
(Aviation value)
 Set iteration counter K = 0, and select an
initial solution 𝑥 ∈ 𝑋
 Set 𝑥′′
= 𝑥
 Start the process as per flow chart by
generating a random set of trial solutions
S(𝑥) ∈ N(𝑥)

Fuzzy Optimization (1/4)
3/22/2018
30
(0) (1)(0-1)
• Fuzzy (Vagueness): For cold state
value assigned as zero (𝑥= 0) and for hot
state one (𝑥 =1).
• Vagueness arises for warm state, value lies
of which between zero to one {𝑥 ∈ (0,1)}
• Concepts involve to define fuzziness are
often difficult to model
• fuzzy can be used when:
 One or more control variable is constant
 System has uncertainties in either the input
or definition
 Mathematical model of any process does not
exist or difficult to model or too-complex to
evaluate fast enough for real-time operation
or too costly to design
Fuzziness
• Some Important term:
 Singletons: Male-Female, Dead-
Alive (Neither flexibility nor
interval)
 Fuzzy number: Around 80℃, bigger
than 25, etc.
 Fuzzy set: Continuous set [0 to 1],
[Red to yellow]
 Fuzzy linguistic term: Beautiful
lady, good quality etc.

• Characteristic function of Fuzzy term
Control variable: A variable that controls the state of the solution
variables
Defuzzification: Process of converting an output fuzzy set into a
single value for a solution variable
Overlap: The degree to which one domain of fuzzy set overlaps to
another
Solution fuzzy set: A temporary fuzzy set created by fuzzy model as
current solution variable
Solution variable: The variable whose value in the fuzzy logic
system is meant to be find
Fuzzy model: A fuzzy system that coverts input into their fuzzy
representations
3/22/2018
31

There are four steps need to be followed to design a fuzzy model
• First step: Define the model function and operation characteristics
• Second step: Define the control surfaces (fuzzy set) using given r concepts
A label meant to be created associated with variable should be an odd number
Each label should overlap (10-50%) with its neighborhood for smooth stable fuzzy
control surface
Density of fuzzy should be highest around optimal control
• Third step: Define the control behavior [if <fuzzy proposition> then <
fuzzy proposition >]
• Fourth step: Select a method of defuzzification (composite maximum,
calculation of concentration)
• Fifth step: fuzzy sets and membership
• Sixth step: Membership functions
3/22/2018
32

• Membership function:
′𝑥′ a control fuzzy variable such that
𝑥 ∈ 𝑋, X is a classical set of objects,
then membership (𝜇 𝐴) of 𝑥 from X
can be understood as
𝜇 𝐴 (𝑥) =
1
0
𝑖𝑓𝑓 𝑥 ∈ 𝐴
𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
Where {0,1} is called a valuation set, if
this set is real Ã is called fuzzy set
𝜇 𝐴 (𝑥) is the degree of membership of
𝑥 in A, the 𝜇 𝐴 (𝑥) more closer to 1 the
more 𝑥 belongs to A
Ã= {(𝑥, 𝜇 𝐴 (𝑥) |𝑥 ∈ 𝑋 )}
3/22/2018
33
Example:
Objective function
Minimize Z= 𝑓(𝑥)
Subjected to: 𝑔𝑗 𝑥 = 0
∀ 𝑗 𝜖 1, … , 𝑝
𝑥 ≥ 0
𝜇 𝑍( 𝑥) = max 𝑥∈𝐴,𝐵{min (𝜇 𝐴
(𝑥), 𝜇 𝐵(𝑥)}
𝑥 is optimal value
{~}defines the fuzziness

Software Availability
There are variety of software available out their as per the problem
type some renown are
• MatLab
• SCOPS, NLCOS (for non-linear)
• MCDA package for R (for multi-criteria-decision making)
• SAMPLE (for uncertainty problem)
• MAPLE (with optimization package)
• Lindo (for LP)
• LGO
Note: Other softwares are also available, user can prefer any as per
their compatibility
3/22/2018
34

References
• X. Liu, “Network optimization with stochastic traffic flows,” Int. J.
Netw. Manag., vol. 12, no. 4, pp. 225–234, 2002.
• T. Ghose, “Optimization Techniques and an Introduction To Genetic
Algorithms and Simulated Annealing,” thesis GA SA 19, no. X, pp. 1–
19, 2002.
• https://en.wikipedia.org/wiki/Fuzzy_logic
• R. V. Rao and V. J. Savsani, Mechanical Design Optimization Using
Advanced Optimization Techniques. 2012.
• C. R. Dyer, “Informed Search.”
• https://neos-guide.org/content/optimization-taxonomy
• S. A.-H. Soliman and A.-A. H. Mantawy, Modern Optimization
Techniques with Applications in Electric Power Systems. 2012.
• http://www.optimization-online.org/cgi-bin/search.cgi
3/22/2018
35

References
• S. E. A. Stephen and J. O. E. A. a, “Review of Ten Non-Traditional
Optimization Techniques,” vol. 3, no. 1, pp. 103–124, 2013.
• S. Mishra, “Chapter 4 Optimization Techniques in Perspective,”
Shodh Ganga a resevoir Indian theses, pp. 26–36, 2013.
• S. H. Z. Edwin K.P Chong, An introduction to optimization, Second
Edition. New Yark/Toronto: Publication, A wiley-Interscience.
• S. Kiranyaz, T. Ince, and M. Gabbouj, Multidimensional Particle
Swarm Optimization for Machine Learning and Pattern Recognition,
vol. 15. 2014.
• https://images.google.com.
• R. Subramani and C. Vijayalakshmi, “A review on advanced
optimization techniques,” ARPN J. Eng. Appl. Sci., vol. 11, no. 19, pp.
11675–11683, 2016.
3/22/2018
36

3/22/2018
37

Introduction to optimization technique

More Related Content

What's hot (20)

Similar to Introduction to optimization technique (20)

Recently uploaded (20)

Introduction to optimization technique