Optimization Introduction power point presentation

EE/Econ 458
Introduction to Optimization
J. McCalley
1

Real-time
Electricity markets and tools
Day-ahead
SCUC and SCED SCED
Minimize f(x)
subject to
h(x)=c
g(x)< b
BOTH LOOK LIKE THIS
SCUC:
x contains
discrete &
continuous
variables.
SCED:
x contains only
continuous
variables.
2

Optimization Terminology
Minimize f(x)
subject to
h(x)=c
g(x)> b
f(x): Objective function
x: Decision variables
h(x)=c: Equality constraint
g(x)> b: Inequality constraint
An optimization problem or a mathematical program
or a mathematical programming problem.
x*: solution 3

Classification of Optimization Problems
http://www.neos-guide.org/NEOS/index.php/Optimization_Tree
Continuous Optimization
Unconstrained Optimization
Bound Constrained Optimization
Derivative-Free Optimization
Global Optimization
Linear Programming
Network Flow Problems
Nondifferentiable Optimization
Nonlinear Programming
Optimization of Dynamic Systems
Quadratic Constrained Quadratic Programming
Quadratic Programming
Second Order Cone Programming
Semidefinite Programming
Semiinfinite Programming
Discrete and Integer Optimization
Combinatorial Optimization
Traveling Salesman Problem
Integer Programming
Mixed Integer Linear Programming
Mixed Integer Nonlinear Programming
Optimization Under Uncertainty
Robust Optimization
Stochastic Programming
Simulation/Noisy Optimization
Stochastic Algorithms
Complementarity Constraints and
Variational Inequalities
Complementarity Constraints
Game Theory
Linear Complementarity Problems
Mathematical Programs with
Complementarity Constraints
Nonlinear Complementarity
Problems
Systems of Equations
Data Fitting/Robust Estimation
Nonlinear Equations
Nonlinear Least Squares
Systems of Inequalities
Multiobjective Optimization
4

Convex functions
Definition #1: A function f(x) is convex in an interval if its
second derivative is positive on that interval.
Example: f(x)=x2
is convex since f’(x)=2x, f’’(x)=2>0
5

Convex functions
The second derivative test is sufficient but not necessary.
www.ebyte.it/library/docs/math09/AConvexInequality.html
Definition #2: A function f(x) is convex if a line drawn
between any two points on the function remains on or
above the function in the interval between the two points.
6

Convex functions
Definition #2: A function f(x) is convex if a line drawn
between any two points on the function remains on or
above the function in the interval between the two points.
Is a linear function convex?
Answer is “yes” since a line drawn between any two
points on the function remains on the function. 7

Convex Sets
Definition #3: A set C is convex if a line segment between
any two points in C lies in C.
Ex: Which of the below are convex sets?
The set on the left is convex. The set on the right is not.
8

Convex Sets
Definition #3: A set C is convex if a line segment between
any two points in C lies in C.
S. Boyd and L. Vandenberghe, “Convex optimization,” Cambridge University Press, 2004.
9

Global vs. local optima
Example: Solve the following:
Minimize f(x)=x2
Solution: f’(x)=2x=0x*=0.
This solution is a local optimum.
It is also the global optimum.
Minimize f(x)=x3
-17x2
+80x-100
Solution: f’(x)=3x2
-34x+80=0
Solving the above results in x=3.33 and x=8.
Issue#1: Which is the best solution?
Issue#2: Is the best solution the global solution?
10

Global vs. local optima
Minimize f(x)=x3
-17x2
+80x-100
Solution: f’(x)=3x2
-34x+80=0. Solving results in x=3.33, x=8.
Issue#1: Which is the
best solution?
Issue#2: Is the best
solution the global
solution?
x=8
No! It is unbounded.
11

Convexity & global vs. local optima
When minimizing a function, if we want to be sure that we
can get a global solution via differentiation, we need to
impose some requirements on our objective function.
We will also need to impose some requirements on the
feasible set S (set of possible values the solution x* may take).
Min f(x)
subject to
h(x)=c
g(x)> b
S
x
x
f

subject to
)
(
min
Definition: If f(x) is a convex function, and if S is a convex set, then
the above problem is a convex programming problem.
Definition: If f(x) is not a convex function, or if S is not a convex set,
then the above problem is a non-convex programming problem. 12
Feasible set

Convex vs. nonconvex programming problems
The desirable quality of a convex
programming problem is that any locally
optimal solution is also a globally optimal
solution. If we have a method of
finding a locally optimal solution, that
method also finds for us the globally
optimum solution.
13
The undesirable quality of a non-convex
programming problem is that any
method which finds a locally optimal
solution does not necessarily find the
globally optimum solution.
MATHEMATICAL
PROGRAMMING
Convex
Non-convex
We address convex
programming
problems in
addressing linear
programming.
We will also, later,
address a special form of
non-convex programming
problems called integer
programs.

A convex programming problem
14
c
x
x
h
s.t.
x
x
f

)
,
(
)
,
(
min
2
1
2
1
c
)
x
h(
x
f

s.t.
)
(
min
c
)
x
(
h
x
f

s.t.
)
(
min
Two variables with one
equality-constraint
Multi-variable with one
equality-constraint.
Multi-variable with multiple
equality-constraints.
We focus on this one, but
conclusions we derive will also
apply to the other two. The
benefit of focusing on this one is
that we can visualize it.

Contour maps
15
Definition: A contour map is a 2-dimensional plane, i.e., a
coordinate system in 2 variables, say, x1, x2, that illustrates
curves (contours) of constant functional value f(x1, x2).
Example: Draw the contour map for
2
2
2
1
2
1 )
,
( x
x
x
x
f 

.
[X,Y] = meshgrid(-
2.0:.2:2.0,-2.0:.2:2.0);
Z = X.^2+Y.^2;
[c,h]=contour(X,Y,Z);
clabel(c,h);
grid;
xlabel('x1');
ylabel('x2');

Contour maps and 3-D illustrations
16
Example: Draw the 3-D surface for
2
2
2
1
2
1 )
,
( x
x
x
x
f 

.
[X,Y] = meshgrid(-
2.0:.2:2.0,-2.0:.2:2.0);
Z = X.^2+Y.^2;
surfc(X,Y,Z)
xlabel('x1')
ylabel('x2')
zlabel('f(x1,x2)')
Height is f(x)
Contours
Each contour of fixed value f
is the projection onto the x1-
x2 plane of a horizontal slice
made of the 3-D figure at a
value f above the x1-x2 plane.

Solving a convex program: graphical analysis
17
Example: Solve this convex program:
.
6
)
,
(
min
2
1
2
1
2
2
2
1
2
1





x
x
)
,x
h(x
s.t.
x
x
x
x
f
6
6 1
2
2 




 x
x
x
x1
Superimpose this
relation on top of the
contour plot for f(x1,x2).
1. f(x1,x2) must be minimized, and so we would like the solution
to be as close to the origin as possible;
2. The solution must be on the thick line in the right-hand corner
of the plot, since this line represents the equality constraint.
A straight line is a convex set because a line
segment between any two points on it remain on it.

18
.
)
25
.
1
,
25
.
1
(
)*
,
(
* 2
1 
 x
x
x
3
)*
,
( 2
1 
x
x
f
Solution:
Any contour f<3 does not
intersect the equality
constraint;
Any contour f>3 intersects the
equality constraint at two
points.
The contour f=3 and the
equality constraint just touch
each other at the point x*.
“Just touch”:
The two curves are tangent to one another at the solution point.

19
.
 The normal (gradient)
vectors of the two curves, at
the solution (tangent) point,
are parallel.
“Parallel” means that the two vectors have the same direction. We do not know that
they have the same magnitude. To account for this, we equate with a “multiplier” λ:
 
c
)
,x
h(x
x
x
f 


 *
2
1
*
2
1 )
,
( 
 
 
*
*
2
2
1
*
2
1
*
2
2
2
1
2
1
1
1
6
}
6
*
)
,
(
{
2
2
)*}
,
(
{
























x
x
x
x
h
x
x
x
x
x
x
f
1
This means the following
two vectors are parallel:

20
.
 
c
)
,x
h(x
x
x
f 


 *
2
1
*
2
1 )
,
( 
  0
)
,
( *
2
1
*
2
1 



 c
)
,x
h(x
x
x
f 
Moving everything to the left:
  0
)
,
( *
2
1
*
2
1 



 )
,x
h(x
c
x
x
f 
Alternately:
 
 
 
 



























0
0
)
,
(
)
,
(
*
2
1
2
1
2
2
1
2
1
1
c
)
,x
h(x
x
x
f
x
c
)
,x
h(x
x
x
f
x


Performing the gradient operation (taking
derivatives with respect to x1 and x2) :
In this problem, we already know the solution, but what if we did not?
Then could we use the above equations to find the solution?

Solving a convex program: analytical analysis
21
 
 
 
 



























0
0
)
,
(
)
,
(
*
2
1
2
1
2
2
1
2
1
1
c
)
,x
h(x
x
x
f
x
c
)
,x
h(x
x
x
f
x


In this problem, we already know the solution, but what if we did not?
Then could we use the above equations to find the solution?
NO! Because we only have 2 equations, yet 3 unknowns: x1, x2, λ.
So we need another equation. Where do we get that equation?
Recall our equality constraint: h(x1, x2)-c=0 . This must be satisfied!
Therefore:
 
 
 
 




































0
0
0
)
,
(
)
,
(
)
,
(
*
2
1
2
1
2
1
2
2
1
2
1
1
c
x
x
h
c
)
,x
h(x
x
x
f
x
c
)
,x
h(x
x
x
f
x


Three equations,
three unknowns,
we can solve.

Solving a convex program: analytical analysis
22
Observation: The three equations are simply partial derivatives of the
function
This is obviously true for the first two equations , but it is not so
obviously true for the last one. But to see it, observe
 
 
 




































0
0
0
)
,
(
)
,
(
)
,
(
*
2
1
2
1
2
1
2
2
1
2
1
1
c
x
x
h
c
)
,x
h(x
x
x
f
x
c
)
,x
h(x
x
x
f
x


 
c
)
,x
h(x
x
x
f 
 2
1
2
1 )
,
( 
 
 
c
)
,x
h(x
c
)
,x
h(x
c
)
,x
h(x
x
x
f











2
1
2
1
2
1
2
1
0
0
)
,
( 


Formal approach to solving our problem
23
Define the Lagrangian function:
In a convex programming problem, the “first-order conditions” for
finding the solution is given by
 
c
)
,x
h(x
x
x
f
x
x 

 2
1
2
1
2
1 )
,
(
)
,
,
( 

L
0
)
,
,
( 2
1 
 
x
x
L
0
)
,
,
(
0
)
,
,
(
0
)
,
,
(
2
1
2
1
2
2
1
1













x
x
x
x
x
x
x
x
L
L
L
OR
Or more
compactly
0
)
,
(
0
)
,
(









x
x
x
L
L
where we have
used x=(x1, x2)

Applying to our example
24
 
 
6
)
,
(
)
,
,
(
2
1
2
2
2
1
2
1
2
1
2
1








x
x
x
x
c
)
,x
h(x
x
x
f
x
x



L
0
)
,
,
( 2
1 
 
x
x
L
0
)
,
,
(
0
)
,
,
(
0
)
,
,
(
2
1
2
1
2
2
1
1













x
x
x
x
x
x
x
x
L
L
L
OR
 
x
x
x
x
x
x
x
x
x
x
x
x
0
6
)
,
,
(
0
2
)
,
,
(
0
2
)
,
,
(
2
1
2
1
2
2
1
2
1
2
1
1























L
L
L
A set of 3 linear equations and 3 unknowns;
we can write in the form of Ax=b.

Applying to our example
25















































































4495
.
2
2247
.
1
2247
.
1
6
0
0
0
1
1
1
2
0
1
0
2
6
0
0
0
1
1
1
2
0
1
0
2
1
2
1
2
1


x
x
x
x

Now, let’s go back to our example with a
nonlinear equality constraint.

Example with nonlinear equality
27
Non-convex because a line connecting two points in the
set do not remain in the set. (see “notes” of this slide)
.
3
2
)
,
(
min
2
1
2
1
2
2
2
1
2
1




x
x
)
,x
h(x
s.t.
x
x
x
x
f
1
2
2
1
2
3
3
2
x
x
x
x 


Superimpose this
relation on top of the
contour plot for f(x1,x2).
1. f(x1,x2) must be minimized, and so we would like the solution
to be as close to the origin as possible;
2. The solution must be on the thick line in the right-hand corner
of the plot, since this line represents the equality constraint.

28
.
)
25
.
1
,
25
.
1
(
)*
,
(
* 2
1 
 x
x
x
3
)*
,
( 2
1 
x
x
f
Solution:
Any contour f<3 does not
intersect the equality
constraint;
Any contour f>3 intersects the
equality constraint at two
points.
The contour f=3 and the
equality constraint just touch
each other at the point x*.
“Just touch”:

29
.
 The normal (gradient)
vectors of the two curves, at
the solution (tangent) point,
are parallel.
“Parallel” means that the two vectors have the same direction. We do not know that
they have the same magnitude. To account for this, we equate with a “multiplier” λ:
 
c
)
,x
h(x
x
x
f 


 *
2
1
*
2
1 )
,
( 
 
 
*
1
2
*
2
2
1
*
2
1
*
2
2
2
1
2
1
2
2
3
2
}
3
*
)
,
(
{
2
2
)*}
,
(
{























x
x
x
x
x
x
h
x
x
x
x
x
x
f
1
This means the following
two vectors are parallel:

30
 
 
 
 



























0
0
)
,
(
)
,
(
*
2
1
2
1
2
2
1
2
1
1
c
)
,x
h(x
x
x
f
x
c
)
,x
h(x
x
x
f
x


This gives us the following two equations.
And we add the equality constraint to give 3 equations, 3 unknowns:
 
 
 
 




































0
0
0
)
,
(
)
,
(
)
,
(
*
2
1
2
1
2
1
2
2
1
2
1
1
c
x
x
h
c
)
,x
h(x
x
x
f
x
c
)
,x
h(x
x
x
f
x


Three equations,
three unknowns,
we can solve.

31
 
 
3
2
)
,
(
)
,
,
(
2
1
2
2
2
1
2
1
2
1
2
1







x
x
x
x
c
)
,x
h(x
x
x
f
x
x



L
0
)
,
,
( 2
1 
 
x
x
L
0
)
,
,
(
0
)
,
,
(
0
)
,
,
(
2
1
2
1
2
2
1
1













x
x
x
x
x
x
x
x
L
L
L
OR
  0
3
2
)
,
,
(
0
2
2
)
,
,
(
0
2
2
)
,
,
(
2
1
2
1
1
2
2
1
2
2
1
2
1
1
















x
x
x
x
x
x
x
x
x
x
x
x
x
x






L
L
L
You can solve this algebraically to obtain
2247
.
1
2
3
2
1 

x
x
2247
.
1
2
3
2
1 



x
x
and f=3 in
both cases

Our approach worked in this case, i.e., we
found a local optimal point that was also a
global optimal point, but because it was
not a convex programming problem, we
had no guarantee that this would happen.
The conditions we established, below, we call first order conditions.
For convex programming problems, they are first order sufficient conditions to provide
the global optimal point.
For nonconvex programming problems, they are first order necessary conditions to
provide the global optimal point.
0
)
,
(
0
)
,
(









x
x
x
L
L

Multiple equality constraints
33
We assume that f and
h are continuously
differentiable.
c
x
h
s.t.
x
f

)
(
)
(
min
   
 
m
m
m c
)
x
(
h
c
)
x
(
h
c
)
x
(
h
x
f
x












...
)
(
)
,
( 2
2
2
1
1
1
L
First order necessary conditions that (x*, λ*) solves the above:
0
)
,
(
0
)
,
(
*
*
*
*









x
x
x
L
L

Multiple equality & 1 inequality constraint
34
We assume that f, h, and
g are continuously
differentiable.
Solution approach:
•Ignore the inequality constraint and solve the problem.
(this is just a problem with multiple equality constraints).
•If inequality constraint is satisfied, then problem is solved.
•If inequality constraint is violated, then the inequality constraint
must be binding  inequality constraint enforced with equality:
b
x
g
c
)
x
(
h
s.t.
x
f


)
(
)
(
min
b
x
g 
)
(
Let’s look at this new problem where the inequality is
binding.

35
g are continuously
differentiable.
b
x
g
c
)
x
(
h
s.t.
x
f


)
(
)
(
min
   
   
b
x
g
c
)
x
(
h
c
)
x
(
h
c
)
x
(
h
x
f
x
m
m
m 









)
(
...
)
(
)
,
,
( 2
2
2
1
1
1






L
First order necessary conditions that (x*, λ*, μ*) solves the above:
0
*)
,
,
(
0
*)
,
,
(
0
)
,
,
(
*
*
*
*
*
*
*

















x
x
x
x
L
L
L
We were able to write down this
solution only after we knew the
inequality constraint was binding.
Can we generalize this approach?

36
   
   
b
x
g
c
)
x
(
h
c
)
x
(
h
c
)
x
(
h
x
f
x
m
m
m 









)
(
...
)
(
)
,
,
( 2
2
2
1
1
1






L
If inequality is not binding, then apply first order necessary
conditions by ignoring it:
μ=0
g(x)-b≠0 (since it is not binding!)
If inequality is binding, then apply first order necessary
conditions treating inequality constraint as an equality constraint
μ≠0
g(x)-b≠0 (since it is binding!)
Either way:
μ(g(x)-b)=0
This relation encodes our
solution procedure!
It can be used to generalize our
necessary conditions

Multiple equality & multiple inequality constraints
37
g are continuously
differentiable.
b
x
g
c
)
x
(
h
s.t.
x
f


)
(
)
(
min
     
     
1
1
1
1
2
1
1
1
2
2
2
1
1
1
)
(
...
)
(
)
(
...
)
(
)
,
,
(
b
x
g
b
x
g
b
x
g
c
)
x
(
h
-
c
)
x
(
h
c
)
x
(
h
x
f
x
n
m
m
m






















L
First order necessary conditions that (x*, λ*, μ*) solves the above:
k
k
b
x
g
x
x
x
x
k
k
k














0
0
)
)
(
(
0
*)
,
,
(
0
*)
,
,
(
0
)
,
,
(
*
*
*
*
*
*
*
*
*
*










L
L
L
Complementarity condition:
Inactive constraints have a
zero multiplier.
Nonnegativity
on inequality
multipliers.
These conditions also referred
to as the Kurash-Kuhn-
Tucker (KKT) conditions

An additional requirement
38
g are continuously
differentiable.
b
x
g
c
)
x
(
h
s.t.
x
f


)
(
)
(
min
For KKT to guarantee finds a local optimum, we need the Kuhn-
Tucker Constraint Qualification (even under convexity).
This condition imposes a certain restriction on the constraint functions .
Its purpose is to rule out certain irregularities on the boundary of the
feasible set, that would invalidate the Kuhn-Tucker conditions should the
optimal solution occur there.
We will not try to tackle this idea, but know this:
If the feasible region is a convex set formed by linear
constraints only, then the constraint qualification will be met, and
the Kuhn-Tucker conditions will always hold at an optimal
solution.

Generator unit cost function:
COSTi =
where
COSTi = production cost
Pi = production power
Economic dispatch calculation (EDC)
    i
i
i
i
i
i
i c
P
b
P
a
P
C 


2
Unit capacity limits
i
i P
P i
i P
P 
level
generation
P
P
level
generation
P
P
where
i
i
max
min
:
max
min




Notation: double underline means lower bound.
Double overline means upper bound.
T
tie
LOSS
D
n
i
i P
P
P
P
P 





1
Power balance
(no transmission
representation)

 



n
i
i
i
i P
C
P
f
1
)
(
min
Subject to
i
i
i
i
i
i
T
n
i
i
P
P
P
P
P
P
P
P









1
General EDC problem statement.
     
 
   
   
2
2
2
2
2
2
1
1
1
1
1
1
2
1
2
2
1
1
2
2
1
1
2
1 ,
,
,
,
,
,
P
P
P
P
P
P
P
P
P
P
P
P
C
P
C
P
P
T

























L
Two unit system, Lagrangian function:
 
 
 
     
   
2
,
1
,
0
2
,
1
,
0
0
,
0
0
,
0
0
0
0
0
0
0
0
2
2
2
2
2
2
1
1
1
1
1
1
2
1
2
2
2
2
2
2
1
1
1
1
1
1













































k
k
P
P
P
P
P
P
P
P
c
x
h
P
P
P
P
P
C
P
P
P
C
P
k
k
T














L
L
L
Two unit system, KKT conditions:

Unit 1 Unit 2
Generation
Specifications:
Minimum Generation 200 MW 100 MW
Maximum Generation 380 MW 200 MW
Cost Curve Coefficients:
Quadratic Term 0.016 0.019
Linear Term 2.187 2.407
Constant Term 120.312 74.074
   
   
100
,
200
200
100
200
,
380
380
200
400
)
,
(
074
.
74
407
.
2
019
.
0
)
(
312
.
120
187
.
2
016
.
0
)
(
2
2
2
1
1
1
2
1
2
1
2
2
2
2
2
1
2
1
1
1























P
P
P
P
P
P
P
P
P
P
h
P
P
P
C
P
P
P
C

     
   
 
   
   
200
100
380
200
400
074
.
74
407
.
2
019
.
0
312
.
120
187
.
2
016
.
0
,
,
,
,
,
,
2
2
2
2
1
1
1
1
2
1
2
2
2
1
2
1
2
2
1
1
2
1



















P
P
P
P
P
P
P
P
P
P
P
P










L
LaGrangian function
 
 
 
     
    0
200
,
0
100
0
380
,
0
200
0
0
400
0
0
407
.
2
038
.
0
0
0
187
.
2
032
.
0
0
2
2
2
2
1
1
1
1
2
1
2
2
2
2
1
1
1
1







































P
P
P
P
c
x
g
P
P
P
P
P
P












L
L
L
KKT conditions

Assume all inequality constraints are non-binding.
This means that
n
i
and i
i ,
1
0
0 


 

 
 
0
400
0
407
.
2
038
.
0
0
187
.
2
032
.
0
2
1
2
1









P
P
P
P


And KKT conditions become
   
   
400
407
.
2
038
.
0
187
.
2
0
032
.
0
2
1
2
1
2
1










P
P
P
P
P
P


Rewrite them as:
And it is easy to see
how to put them
into matrix form for
solution in matlab.





































400
407
.
2
187
.
2
0
1
1
1
038
.
0
0
1
0
032
.
0
2
1

P
P
Solution yields:





















24
.
9
71
.
179
29
.
220
2
1

P
P

What is  = $9.24/MW-hr ???
It is the system “incremental cost.”
It is the cost if the system provides an
additional MW over the next hour.
It is the cost of “increasing” the RHS of the
equality constraint by 1 MW for an hour.
We can verify this.

Verification for meaning of lambda.
• Compute total costs/hr for Pd=400 MW
• Compute total costs/hr for Pd=401 MW
• Find the difference in total costs/hr for
the two demands.
If our interpretation of lambda is correct,
this difference should be $9.24.

     
 
     
  hr
P
C
P
C
hr
P
C
P
C
/
$
25
.
1120
074
.
74
71
.
179
407
.
2
71
.
179
019
.
0
/
$
53
.
1378
312
.
120
29
.
220
187
.
2
29
.
220
016
.
0
2
2
2
2
2
1
1
2
1
1








    78
.
2498
25
.
1120
53
.
1378
2
2
1
1 



 P
C
P
C
CT
Total cost/hr are C1+C2
Get cost/hr for each unit.

Now solve EDC for Pd=401 MW to get P1,P2




































401
407
.
2
187
.
2
0
1
1
1
038
.
0
0
1
0
032
.
0
2
1

P
P





















25
.
9
17
.
180
83
.
220
2
1

P
P

     
 
     
  hr
P
C
P
C
hr
P
C
P
C
/
$
51
.
1124
074
.
74
17
.
180
407
.
2
17
.
180
019
.
0
/
$
52
.
1383
312
.
120
83
.
220
187
.
2
83
.
220
016
.
0
2
2
2
2
2
1
1
2
1
1








Total cost/hr are C1+C2
Get cost/hr for each unit.
    03
.
2508
51
.
1124
52
.
1383
2
2
1
1 



 P
C
P
C
CT
Total cost/hr changed by 2508.03-2498.78 = 9.25 $/hr,
which is in agreement with our interpretation of lambda.

Optimization Introduction power point presentation

More Related Content

Similar to Optimization Introduction power point presentation (20)

Recently uploaded (20)

Optimization Introduction power point presentation

Editor's Notes