Derivative free optimizations
0 likes645 views
This document discusses derivative-free optimization techniques. It begins with an overview of optimization problems, including single-objective, many-objective, convex, and non-convex problems. It then discusses the limitations of non-convex problems and solutions for convex problems using gradient-based techniques. The document focuses on derivative-free optimization, providing examples of techniques like simulated annealing, genetic algorithms, and particle swarm optimization that do not require derivative information. It concludes with references for further reading.
![Convex and Non-convex Optimization
Algebraically, f is convex if, for any x and y, and
any t between 0 and 1, f( t*x + (1-t)*y ) <= t*f(x)
+ (1-t)*f(y). A function is concave if -f is convex,
i.e., if the chord from x to y lies on or below the
graph of f. [*=multiplication]
Rajdeep Chatterjee, KIIT-DU 9](https://image.slidesharecdn.com/derivative-freeoptimizations-210928124620/85/Derivative-free-optimizations-9-320.jpg)
Derivative free optimizations
- 1. Derivative-free Optimization CI (CS-3030) Rajdeep Chatterjee Assistant Professor School of Computer Engineering KIIT Deemed to be University, Bhubaneswar-751024, Odisha, India
- 2. Agenda • Optimization problems • Types of problems • Single-objective optimization • Many-objective optimization • Convex optimization problem • Convex and non-convex optimization • Limitations of non-convex optimization • Solutions for convex optimization • Gradient-based convex optimization • Derivative-free optimization and its properties • Examples of derivative-free optimization techniques Rajdeep Chatterjee, KIIT-DU 2
- 3. • Mathematical optimization (alternatively spelled optimisation) or mathematical programming is the selection of a best element (with regard to some criterion) from some set of available alternatives. • An optimization problem consists of maximizing or minimizing a real function by systematically choosing input values from within an allowed set and computing the value of the function. Rajdeep Chatterjee, KIIT-DU 3
- 4. • Type of problems, such as and (Generally, focus mainly is on non-linear optimization problems, which are far more complex problems.) • Nonlinear optimization is the process of solving an optimization problem where some of the constraints or the objective function are nonlinear. Rajdeep Chatterjee, KIIT-DU 4
- 5. • Type of problems, , when we just need to focus on the search of one solution; , when there is more than one optimal solution for a problem; , when the problem has a set of solutions and we need to find a diverse set; § Multi-objective problems having more than three objectives are referred to as optimization problems; and , where the conditions for a problem change over time and there is need to track the optima. Rajdeep Chatterjee, KIIT-DU 5
- 6. objective optimization problem, • x is a vector of n decision variables: x = (x1, x2, ..., xn)T • Each decision variable xi to take a value within a lower x(L) i and an upper x(U) i bounds. • Associated with the problem are J inequality and K equality constraints and the terms gj(x) and hk(x) are called constraint functions. • Although the inequality constraints are treated as ≥ types, the ≤ constraints can also be considered in the above formulation by converting those to ≥ types simply by multiplying each constraint function by 1. Rajdeep Chatterjee, KIIT-DU 6
- 7. objective optimization problem, • In many-objective optimization, the M objective functions f(x) = (f1(x), f2(x), ..,fM(x))T • Many optimization algorithms are developed to solve only one type of optimization problems e.g., minimization problems. • Otherwise, an objective is required to be maximized by using such an algorithm, the duality principle can be used to transform the original objective for maximization into an objective for minimization by multiplying objective function by 1. Rajdeep Chatterjee, KIIT-DU 7
- 8. Convex Optimization Problem • The following are useful properties of convex optimization problems: § Every local minimum is a global minimum; § The optimal set is convex; § If the objective function is strictly convex, then the problem has at most one optimal point. e.g., Linear programming. Rajdeep Chatterjee, KIIT-DU 8
- 9. Convex and Non-convex Optimization Algebraically, f is convex if, for any x and y, and any t between 0 and 1, f( t*x + (1-t)*y ) <= t*f(x) + (1-t)*f(y). A function is concave if -f is convex, i.e., if the chord from x to y lies on or below the graph of f. [*=multiplication] Rajdeep Chatterjee, KIIT-DU 9
- 10. Limitations of Non-convex Problem • Many local minimas; • Plateaux (plural): A plateau is encountered when the search space is flat, or sufficiently flat that the value returned by the target function is indistinguishable from the value returned for nearby regions • Saddle points: In mathematics, a saddle point or minimax point is a point on the surface of the graph of a function where the slopes in orthogonal directions are all zero, but which is not a local extremum of the function. Rajdeep Chatterjee, KIIT-DU 10
- 11. Solutions for Convex Optimization Problem • Gradient-based optimization techniques, capable of determining search directions according to an objective function's derivative information. • Gradient descent is a first-order iterative optimization algorithm for finding a local minimum of a differentiable function. § Steepest descent method (also known as Gradient descent); § Newton's method; § Conjugate gradient descent method, etc. Rajdeep Chatterjee, KIIT-DU 11
- 12. Visualization of Gradient-based Convex Optimization Rajdeep Chatterjee, KIIT-DU 12
- 13. Prerequisite - Partial derivative Rajdeep Chatterjee, KIIT-DU 13
- 14. Prerequisite - Partial derivative Rajdeep Chatterjee, KIIT-DU 14
- 15. Derivative-free optimization • Sometimes information about the derivative of the objective function f is unavailable, unreliable or impractical to obtain. The problem to find optimal points in such situations is referred to as derivative-free optimization. Rajdeep Chatterjee, KIIT-DU 15
- 16. Examples of Derivative-free Optimization • Simulated Annealing, • Genetic Algorithm, • Particle Swarm Optimization, • Differential Evolution, etc. Rajdeep Chatterjee, KIIT-DU 16
- 17. References • AA222: Introduction to Multidisciplinary Design Optimization, Stanford University, USA. • EE364a: Convex Optimization I, Stanford University, USA. • ECE1505: Convex Optimization, University of Toronto, Canada. • R. Tyrell Rockafellar, Convex Analysis, Princeton University Press, USA. • Nonlinear Programming, Convex Programming Problems, NPTEL, India. • 18.02: Multivariable Calculus (Fall 2007), MIT OCW, USA. • 6.079 / 6.975: Introduction to Convex Optimization, MIT OCW, USA. Rajdeep Chatterjee, KIIT-DU 17