Design & Analysis of Algorithm course .pptx

Design and Analysis of
Algorithms

What is an algorithm?
“computer”
• An algorithm is any well-defined computational procedure that takes some value, or set
of values, as input and produces some value, or set of values as output.
OR
• An algorithm is a list of steps (sequence of unambiguous instructions ) for solving a
problem that transforms the input into the output.
OR
• A finite set of instructions that specifies a sequence of operation to be carried out in
order to solve a specific problem is called an algorithm
problem
algorithm
input output
8/25/2020 DAA - Unit - I Presentation Slides 3

Example: gcd(m, n)
Algorithms for the same problem can be based on very different ideas and can solve
the problem with dramatically different speeds.
The greatest common divisor of two nonnegative, not-both-zero integers m and n,
denoted gcd(m, n), is defined as the largest integer that divides
both m and n evenly.
In modern terms, Euclid’s algorithm is based on applying repeatedly the equality
gcd(m, n) = gcd(n, m mod n),
where m mod n is the remainder of the division of m by n, until m mod n is
equal to 0.
Since gcd(m, 0) = m, the last value of m is also the greatest common divisor of
the initial m and n.

For example, gcd(60, 24) can be computed as follows:
gcd(60, 24) = gcd(24, 12) = gcd(12, 0) = 12
Here is a more structured description of this algorithm:
Euclid’s algorithm for computing gcd(m, n)
Step 1 If n = 0, return the value of m as the answer and stop; otherwise, proceed
to Step 2.
Step 2 Divide m by n and assign the value of the remainder to r.
Step 3 Assign the value of n to m and the value of r to n. Go to Step 1.
I. Euclid’s algorithm for computing
gcd(m, n)

Euclid’s algorithm for computing
gcd(m, n) (contd...)
Alternatively, we can express the same algorithm in pseudocode:
ALGORITHM Euclid(m, n)
//Computes gcd(m, n) by Euclid’s algorithm
//Input: Two nonnegative, not-both-zero integers m and n
//Output: Greatest common divisor of m and n
while n = 0 do
r ← m mod n
m ← n
n ← r return m

II. Middle-school procedure for computing gcd(m, n)
Step 1 Find the prime factors of m.
Step 2 Find the prime factors of n.
Step 3 Identify all the common factors in the two prime expansions found in Step
1 and Step 2.
(If p is a common factor occurring pm and pn times in m and n, respectively, it
should be repeated min{pm, pn} times.)
Step 4 Compute the product of all the common factors and return it as
the greatest common divisor of the numbers given.
Thus, for the numbers 60 and 24, we get
60 = 2 . 2 . 3 . 5
24 = 2 . 2 . 2 . 3
gcd(60, 24) = 2 . 2 . 3 = 12.

• Input
• Output
• Definiteness
• Finiteness
• Effectiveness
• Correctness
• Simplicity
• Unambiguous
• Feasibility
• Portable
• Independent
Characteristics of an Algorithm

Apriori and posteriori Analysis
• Apriori analysis means, analysis is performed prior to running it on a specific
system. This analysis is a stage where a function is defined using some theoretical
model.
Hence, we determine the time and space complexity of an algorithm by just looking
at the algorithm rather than running it on a particular system with a different
memory, processor, and compiler.
• Posteriori analysis of an algorithm means we perform analysis of an algorithm only
after running it on a system. It directly depends on the system and changes from
system to system.
• In an industry, we cannot perform Aposteriori analysis as the software is generally
made for an anonymous user, which runs it on a system different from those present
in the industry.
• In Apriori, it is the reason that we use asymptotic notations to determine time and
space complexity as they change from computer to computer; however,
asymptotically they are the same.

Algorithm Expectation
• Correctness:
• Correct: Algorithms must produce correct result.
• Produce an incorrect answer: Even if it fails to give correct results all the time
still there is a control on how often it gives wrong result. Eg. Rabin-Miller
Primality Test
• Approximation algorithm: Exact solution is not found, but near optimal solution
can be found out. (Applied to optimization problem.)
• Less resource usage:
• Algorithms should use less resources (time and space).
• Resource usage:
Here, the time is considered to be the primary measure of efficiency .
We are also concerned with how much the respective algorithm involves the computer memory.
But mostly time is the resource that is dealt with.
The actual running time depends on a variety of backgrounds: like the speed of the Computer,
the language in which the algorithm is implemented, the compiler/interpreter, skill of the
programmers etc.
So, mainly the resource usage can be divided into: 1.Memory (space) 2.Time

Analysis of an algorithm
1. instruction space
Performance
• Two areas are important for performance:
1. space efficiency - the memory required, also called, space complexity
2. time efficiency - the time required, also called time complexity
• Space efficiency:
There are some circumstances where the space/memory used must be analyzed. For
example, for large quantities of data or for embedded systems programming.
• Components of space/memory use:
Affected by: the compiler, compiler
options, target computer (cpu)
Affected by: the data size/dynamically
allocated memory, static program
variables,
Affected by: the compiler, run-time
function calls and recursion, local
variables, parameters
2. data space
3. run-time stack space

Space efficiency
3. run-time stack space
Affected by: the compiler, run-time
function calls and recursion, local
variables, parameters
The space requirement has fixed/static/compile time and a variable/dynamic/runtime
components. Fixed components are the machine language instructions and static
variables. Variable components are the runtime stack usage and dynamically allocated
memory usage.
One circumstance to be aware of is, does the approach require the data to be
duplicated in memory (as does merge sort). If so we have 2N memory use.

Time efficiency
The actual running time depends on many factors:
•The speed of the computer: cpu (not just clock speed), I/O, etc.
•The compiler, compiler options .
•The quantity of data - ex. search a long list or short.
•The actual data - ex. in the sequential search if the name is first or last.
Time Efficiency - Approaches
When analyzing for time complexity we can take two approaches:
1.Order of magnitude/asymptotic categorization - This uses coarse categories and
gives a general idea of performance. If algorithms fall into the same category, if
data size is small, or if performance is critical, then the next approach can be
looked at more closely.
2.Estimation of running time.

Algorithm as a Technology
Algorithms are just like a technology. We all use latest and greatest processors but
we need to run implementations of good algorithms on that computer in order to
properly take benefits of our money that we spent to have the latest processor.
Example:
• A faster computer (computer A) running a sorting algorithm whose running time
on n values grows like n2 against a slower computer (computer B) running a
sorting algorithm whose running time grows like n lg n.
• They each must sort an array of 10 million numbers.

Algorithm as a Technology (Contd..)
• Computer A executes 10 billion instructions per second (faster than any single
sequential computer at the time of this writing)
• Computer B executes only 10 million instructions per second, so that computer A
is 1000 times faster than computer B in raw computing power.
• To make the difference even more dramatic, suppose that the world’s craftiest
programmer codes in machine language for computer A, and the resulting code
requires 2n2 instructions to sort n numbers.
• Suppose further that just an average programmer writes for computer B, using a
high level language with an inefficient compiler, with the resulting code taking
50n lg n instructions.

Algorithm as a Technology (Contd..)
• So choosing a good algorithm (algorithm with slower rate of growth) as used by
computer B affects a lot.
Computer A
Running time grows like n2.
10 billion instructions per sec.
2n2 instruction.
Time taken= 2*
(107)2/1010=20,000
It is more than 5.5hrs
Computer B
Grows in n log n.
10million instruction per sec
50 n log n instruction
(50*107*log 107)/107≈1163
it is under 20 mins.

Faster Algorithm vs. Faster CPU
 A faster algorithm running on a slower machine will always win for large enough
instances
 Suppose algorithm S1 sorts n keys in 2n2 instructions
 Suppose computer C1 executes 1 billion instruc/sec
 When n = 1 million, takes 2000 sec
 Suppose algorithm S2 sorts n keys in 50nlog2n instructions
 Suppose computer C2 executes 10 million instruc/sec
 When n = 1 million, takes 100 sec

Algorithm Design Techniques
The following is a list of several popular design approaches:
1. Divide and Conquer Approach: It is a top-down approach. The algorithms
which follow the divide & conquer techniques involve three steps:
•Divide the original problem into a set of subproblems.
•Solve every subproblem individually, recursively.
•Combine the solution of the subproblems (top level) into a solution of the
whole original problem.
2. Greedy Technique: Greedy method is used to solve the optimization
problem. An optimization problem is one in which we are given a set of input
values, which are required either to be maximized or minimized (known as
objective), i.e. some constraints or conditions.
•Greedy Algorithm always makes the choice (greedy criteria) looks best at the
moment, to optimize a given objective.
•The greedy algorithm doesn't always guarantee the optimal solution however
it generally produces a solution that is very close in value to the optimal.

3. Dynamic Programming: Dynamic Programming is a bottom-up approach we solve
all possible small problems and then combine them to obtain solutions for bigger
problems.
This is particularly helpful when the number of copying subproblems is exponentially
large. Dynamic Programming is frequently related to Optimization Problems.
4. Branch and Bound: In Branch & Bound algorithm a given subproblem, which
cannot be bounded, has to be divided into at least two new restricted subproblems.
Branch and Bound algorithm are methods for global optimization in non-convex
problems. Branch and Bound algorithms can be slow, however in the worst case they
require effort that grows exponentially with problem size, but in some cases we are
lucky, and the method coverage with much less effort.

5. Randomized Algorithms: A randomized algorithm is defined as an
algorithm that is allowed to access a source of independent, unbiased random
bits, and it is then allowed to use these random bits to influence its
computation.
6. Backtracking Algorithm: Backtracking Algorithm tries each possibility until
they find the right one. It is a depth-first search of the set of possible solution.
During the search, if an alternative doesn't work, then backtrack to the choice
point, the place which presented different alternatives, and tries the next
alternative.
7. Randomized Algorithm: A randomized algorithm uses a random number at
least once during the computation make a decision.

Algorithm Design and Analysis Process

Algorithm Analysis
Generally, we perform the following types of analysis −
• Worst-case − The maximum number of steps taken on any instance of size a.
• Best-case − The minimum number of steps taken on any instance of size a.
• Average case − An average number of steps taken on any instance of size a.

Performance measures: worst case,
average case and Best case

L1.24
Running time
• The running time depends on the input: an already sorted sequence is easier to
sort.
• Major Simplifying Convention: Parameterize the running time by the size of
the input, since short sequences are easier to sort than long ones.
TA(n) = time of A on length n inputs
• Generally, we seek upper bounds on the running time, to have a guarantee of
performance.

L1.25
Kinds of analyses
Worst-case: (usually)
• T(n) = maximum time of algorithm on any input of size n.
Average-case: (sometimes)
• T(n) = expected time of algorithm over all inputs of size n.
• Need assumption of statistical distribution of inputs.
Best-case: (NEVER)
• Cheat with a slow algorithm that works fast on some input.

Methodology of Analysis
• Asymptotic Analysis
• The asymptotic behavior of a function f(n) refers to the growth of f(n) as n
gets large.
• We typically ignore small values of n, since we are usually interested in
estimating how slow the program will be on large inputs.
• A good rule of thumb is that the slower the asymptotic growth rate, the
better the algorithm. Though it’s not always true.
• For example, a linear algorithm f(n)=d∗n+k is always asymptotically better
than a quadratic one, f(n)=c. n2+q.

Methodology of Analysis
• Solving Recurrence Equations
• A recurrence is an equation or inequality that describes a function in terms of its
value on smaller inputs. Recurrences are generally used in divide-and-conquer
paradigm.
• Let us consider T(n) to be the running time on a problem of size n.
• If the problem size is small enough, say n < c where c is a constant, the
straightforward solution takes constant time, which is written as θ(1). If the
division of the problem yields a number of sub-problems with size nb
• A recurrence relation can be solved using the following methods −
• Substitution Method − In this method, we guess a bound and using
mathematical induction we prove that our assumption was correct.
• Recursion Tree Method − In this method, a recurrence tree is formed
where each node represents the cost.
• Master’s Theorem − This is another important technique to find the
complexity of a recurrence relation.

Growth of Function (Asymptotic
Notations)
• The complexity of an algorithm describes the efficiency of the algorithm in terms of
the amount of the memory required to process the data and the processing time.
• Complexity of an algorithm is analyzed in two perspectives: Time and Space.
• Execution time of an algorithm depends on the instruction set, processor speed, disk
I/O speed, etc. Hence, we estimate the efficiency of an algorithm asymptotically.
• Time function of an algorithm is represented by T(n), where n is the input size.
• Different types of asymptotic notations are used to represent the complexity of an
algorithm. Following asymptotic notations are used to calculate the running time
complexity of an algorithm.
• O − Big Oh
• Ω − Big omega
• θ − Big theta
• o − Little Oh
• ω − Little omega

Notations)contd..
• O: Asymptotic Upper Bound
• ‘O’ (Big Oh) is the most commonly used notation. A function f(n) can be
represented is the order of g(n) that is O(g(n)), if there exists a value of positive
integer n as n0 and a positive constant c such that −
f(n)⩽c.g(n) for n>n0 in all case
• Hence, function g(n) is an upper bound for function f(n), as g(n) grows faster
than f(n).
Example
• Let us consider a given function, f(n)=4.n3+10.n2+5.n+1
• Considering g(n)=n3
• f(n)⩽5.g(n) for all the values of n>2
• Hence, the complexity of f(n) can be represented as O(g(n)) i.e. O(n3)

Notations)Contd...

Big-O Examples
for n >= 7
5n + 7 <= 5n + n
<= 6n
<= 4n2
So c = 6 and n0 = 7, thus 5n + 7 is O(n).
Example 2
3n2 + 4n <= 3n2 + n2 for n >= 4
So c = 4 and n0 = 4, thus 3n2 + 4n is O(n2).
It is usually assumed the bound is tight. For example, 3n + 4 function is bounded by n2 for n
> 4 and so is O(n2). This is not a tight bound however, 3n + 4 is O(n).
Example 1

Notations)Contd...
• θ: Asymptotic Tight Bound
• We say that f(n)=θ(g(n)) when there exist constants c1 and c2 that
c1.g(n)⩽f(n)⩽c2.g(n)for all sufficiently large value of n. Here n is a positive
integer.
• This means function g is a tight bound for function f.
Example
• Considering g(n)=n3 , 4.g(n)⩽f(n)⩽5.g(n)for all the large values of n.
• Hence, the complexity of f(n) can be represented as θ(g(n)) , i.e. θ(n3).

Notations)Contd...
• Ω: Asymptotic Lower Bound
• We say that f(n)=Ω(g(n))
f(n)=Ω(g(n)) when there exists constant c that f(n)⩾c.g(n),
f(n)⩾c.g(n) for all sufficiently large value of n. Here n is a positive integer. It
means function g is a lower bound for function f; after a certain value of n, f will
never go below g.
Example
• Considering g(n)=n3, f(n)⩾4.g(n) for all the values of n>0.
• Hence, the complexity of f(n) can be represented as Ω(g(n)), i.e. Ω(n3)

1 O(1) constant
log n O(log n) logarithmic
n O(n) linear
n log n O(n log n) n log n
n2 O(n2) quadratic
n3 O(n3) cubic
2n O(2n) exponential
n! O(n!) factorial
Common Asymptotic Notations
Function Big-O Name

Rate of growth of function
Logarithmic Linear Linear
logarithmic
Quadratic Polynomial Exponential
Log2n N nlog2n n2 n3 2n
0 1 0 1 1 2
1 2 2 4 8 4
2 4 8 16 64 16
3 8 24 64 512 256
4 16 64 256 4096 65536
5 32 160 1024 32768 4294967296
3.322 10 33.22 102 103 > 103
6.644 102 664.4 104 106 > >1025
9.966 103 9966.0 106 10 > > 10250
8/25/2020 42
DAA - Unit - I Presentation Slides

• f(n) = O(g(n)) and
g(n) = O(h(n)) ⇒ f(n) = O(h(n))
• f(n) = Ω(g(n)) and
g(n) = Ω(h(n)) ⇒ f(n) = Ω(h(n))
• f(n) = Θ(g(n)) and
g(n) = Θ(h(n)) ⇒ f(n) = Θ(h(n))
• f(n) = O(f(n))
f(n) = Ω(f(n))
f(n) = Θ(f(n))
Comparison of Functions
Reflexivity
Transitivity

• f(n) = Θ(g(n)) ⇐⇒ g(n) = Θ(f(n))
• f(n) = O(g(n)) ⇐⇒ g(n) = Ω(f(n))
• f(n) = o(g(n)) ⇐⇒ g(n) = ω(f(n))
• f(n) = O(g(n)) and
f(n) = Ω(g(n)) ⇒ f(n) = Θ(g(n))
Transpose
Symmetry
Symmetry
Theorem 3.1

Asymptotic Analysis and Limits

• f1(n) = O(g1(n)) and f2(n) = O(g2(n)) ⇒
f1(n) + f2(n) = O(g1(n) + g2(n))
• f(n) = O(g(n)) ⇒ f(n) + g(n) = O(g(n))

Standard Notation and Common
Functions
• Monotonicity
A function f(n) is monotonically increasing if m  n implies f(m)  f(n) .
A function f(n) is monotonically decreasing if m  n implies f(m)  f(n) .
A function f(n) is strictly increasing
if m < n implies f(m) < f(n) .
A function f(n) is strictly decreasing
if m < n implies f(m) > f(n) .

Standard Notation and Common
Functions
• Floors and ceilings
For any real number x, the greatest integer less than or equal to x is denoted by
x.
For any real number x, the least integer greater than or equal to x is denoted by
x.
For all real numbers x,
x1 < x  x  x < x+1.
Both functions are monotonically increasing.

44
Recurrences and Running Time
• An equation or inequality that describes a function in terms of its value on smaller
inputs.
T(n) = T(n-1) + n
• Recurrences arise when an algorithm contains recursive calls to itself
• What is the actual running time of the algorithm?
• Need to solve the recurrence
• Find an explicit formula of the expression
• Bound the recurrence by an expression that involves n

45
Example Recurrences
• T(n) = T(n-1) + n Θ(n2)
• Recursive algorithm that loops through the input to eliminate one item
• T(n) = T(n/2) + c Θ(lg n)
• Recursive algorithm that halves the input in one step
• T(n) = T(n/2) + n Θ(n)
• Recursive algorithm that halves the input but must examine every item in
the input
• T(n) = 2T(n/2) + 1 Θ(n)
• Recursive algorithm that splits the input into 2 halves and does a constant
amount of other work

Analysis of BINARY-SEARCH
Alg.: BINARY-SEARCH (A, lo, hi, x)
constant time: c2
same problem of size n/2
same problem of size n/2
constant time: c1
if (lo > hi)
return FALSE
mid  (lo+hi)/2
if x = A[mid]
return TRUE
if ( x < A[mid] )
BINARY-SEARCH (A, lo, mid-1, x)
if ( x > A[mid] )
BINARY-SEARCH (A, mid+1, hi, x)
• T(n) = c + T(n/2)
• T(n) – running time for an array of size n
constant time: c3
46

47
Methods for Solving Recurrences
• Iteration method
• Substitution method
• Recursion tree method
• Master method

48
The Iteration Method
• Convert the recurrence into a summation and try to bound it using known series
• Iterate the recurrence until the initial condition is reached.
• Use back-substitution to express the recurrence in terms of n and the initial
(boundary) condition.

The Iteration Method
T(n) = c + T
T
(n
(n
//
22
)) = c + T(n/4)
T(n) = c + T(n/2)
= c + c + T(n/4)
= c + c + c + T(n/8)
Assume n = 2k
T(n) = c + c + … + c + T(1)
k times
= clgn + T(1)
= Θ(lgn)
49
T(n/4) = c + T(n/8)

50
Iteration Method – Example
T(n) = n + 2T(n/2)
T(n) = n + 2T(n/2)
= n + 2(n/2 + 2T(n/4))
= n + n + 4T(n/4)
= n + n + 4(n/4 + 2T(n/8))
= n + n + n + 8T(n/8)
… = in + 2iT(n/2i)
= kn + 2kT(1)
= nlgn + nT(1) = Θ(nlgn)
Assume: n = 2k
T(n/2) = n/2 + 2T(n/4)

51
The substitution method
1. Guess a solution
2. Use induction to prove that the solution works

52
Substitution method
• Guess a solution
• T(n) = O(g(n))
• Induction goal: apply the definition of the asymptotic notation
• T(n) ≤ d g(n), for some d > 0 and n ≥ n0
• Induction hypothesis: T(k) ≤ d g(k) for all k < n
• Prove the induction goal
• Use the induction hypothesis to find some values of the
constants d and n0 for which the induction goal holds
(strong induction)

53
Example: Binary Search
T(n) = c + T(n/2)
• Guess: T(n) = O(lgn)
• Induction goal: T(n) ≤ d lgn, for some d and n ≥ n0
• Induction hypothesis: T(n/2) ≤ d lg(n/2)
• Proof of induction goal:
T(n) = T(n/2) + c ≤ d lg(n/2) + c
= d lgn – d + c ≤ d lgn
if: – d + c ≤ 0, d ≥ c

54
Example
T(n) = T(n-1) + n
• Guess: T(n) = O(n2)
• Induction goal: T(n) ≤ c n2, for some c and n ≥ n0
• Induction hypothesis: T(n-1) ≤ c(n-1)2 for all k < n
• Proof of induction goal:
T(n) = T(n-1) + n ≤ c (n-1)2 + n
= cn2 – (2cn – c - n) ≤ cn2
if: 2cn – c – n ≥ 0  c ≥ n/(2n-1)  c ≥ 1/(2 – 1/n)
• For n ≥ 1  2 – 1/n ≥ 1  any c ≥ 1 will work

60
The recursion-tree method
Convert the recurrence into a tree:
• Each node represents the cost incurred at various levels
of recursion
• Sum up the costs of all levels
Used to “guess” a solution for the recurrence

61
Example 1
W(n) = 2W(n/2) + n2
• Subproblem size at level i is: n/2i
• Subproblem size hits 1 when 1 = n/2i  i = lgn
• Cost of the problem at level i = (n/2i)2 No. of nodes at level i = 2i
• Total cost:
2
1
1 1
W (n)    2lg n
W (1)  n2
   n  n2
   O(n) n2
 O(n)  2n2

 1 
i
i0  2 
i0  2 
lg n1
n2 lg n1
 1 
i
i0 2i
 W(n) = O(n2)

62
Example 2
E.g.: T(n) = 3T(n/4) + cn2
• Subproblem size at level i is: n/4i
• Subproblem size hits 1 when 1 = n/4i  i = log4n
• Cost of a node at level i = c(n/4i)2
• Number of nodes at level i = 3i  last level has 3log
4
n = nlog
4
3 nodes
• Total cost:
 T(n) = O(n2)
16
1
16
16
2
2
2
2 log4 3

log4 3
 log4 3
  O(n )
1
  cn n
3
cn   n

 
 3 
i
cn n 
 
 
T(n)  


i0 
log4 n1
 3 
i
i0

Master’s method
• Solving recurrences of the form:
Idea: compare f(n) with nlog a
b
• f(n) is asymptotically smaller or larger than nlog a by a polynomial factor n
b
• f(n) is asymptotically equal with nlog a
b

b

58
 
where, a ≥ 1, b > 1, and f(n) > 0
T(n)  aT n   f (n)

Master’s method
• For solving recurrences of the form:
Case 1: if f(n) = O(nlog a -) for some  > 0, then: T(n) = (nlog a)
b b
Case 2: if f(n) = (nlog a), then: T(n) = (nlog a lgn)
b b
Case 3: if f(n) = (nlog a +) for some  > 0, and if
b
af(n/b) ≤ cf(n) for some c < 1 and all sufficiently large n, then:
T(n) = (f(n))
regularity condition

b

 
where, a ≥ 1, b > 1, and f(n) > 0
T(n)  aT n   f (n)
59

66
Example
T(n) = 2T(n/2) + n
a = 2, b = 2, log22 = 1
Compare nlog 2 with f(n) = n
2
 f(n) = (n)  Case 2
 T(n) = (nlgn)

67
Example
T(n) = 2T(n/2) + n2
a = 2, b = 2, log22 = 1
Compare n with f(n) = n2
 f(n) = (n1+) Case 3  verify regularity cond.
a f(n/b) ≤ c f(n)
 2 n2/4 ≤ c n2  c = ½ is a solution (c<1)
 T(n) = (n2)

68
Example
T(n) = 3T(n/4) + nlgn
a = 3, b = 4, log43 = 0.793
Compare n0.793 with f(n) = nlgn
f(n) = (nlog 3+) Case 3
4
Check regularity condition:
3(n/4)lg(n/4) ≤ (3/4)nlgn = c f(n), c=3/4
T(n) = (nlgn)

Machine-independent time: An
example
A pseudocode for insertion sort ( INSERTION SORT ).
INSERTION-SORT(A)
1 for j  2 to length [A]
2 do key  A[ j]
3  Insert A[j] into the sortted sequence A[1,..., j-1].
4 i  j – 1
5 while i > 0 and A[i] > key
6 do A[i+1]  A[i]
7 i  i – 1
8 A[i +1]  key

Analysis of INSERTION-SORT(contd.)
7
8
6
4
5
2
3
INSERTION -SORT(A)
1 for j  2 to length[A]



cost times
c1 n
c2 n 1
n
j2
n
j2
(t j 1)
(t j 1)
n
j2
t j
do key  A[ j]
 Insert A[ j] into thesorted
sequence A[1 j 1] 0 n 1
i  j 1 c4 n 1
while i  0 and A[i]  key c5
do A[i 1]  A[i] c6
i  i 1 c7
A[i 1] key c8 n 1

j2 j2
The total running time is
n n
T(n)  c1  c2 (n 1)  c4 (n 1)  c5 tj  c6 (tj 1)
n
 c7 (t j 1)  c8 (n 1).
j2

The best case: The array is already sorted.
(tj =1 for j=2,3, ...,n)
T(n)  c1n  c2 (n 1)  c4 (n 1)  c5 (n 1)  c8 (n 1)
 (c1  c2  c4  c5  c8 )n  (c2  c4  c5  c8 ).

•The worst case: The array is reverse sorted
(tj =j for j=2,3, ...,n).
T(n)  c1n  c2 (n 1)  c5 (n(n 1)/ 2 1)
 c6 (n(n 1)/ 2)  c7 (n(n 1)/ 2)  c8 (n 1)
 (c / 2  c / 2  c / 2)n2
 (c  c  c  c / 2  c / 2  c / 2  c )n
5 6 7 1 2 4 5 6 7 8
T(n)  an2  bn  c
2
j 
n(n 1)
n

j1

Shell Sort
• Invented by Donald Shell in 1959.
• 1st algorithm to break the quadratic time barrier but few years later, a sub
quadratic time bound was proven
• Shell sort works by comparing elements that are distant rather than adjacent
elements in an array.
• Shellsort uses a sequence h1, h2, …, ht called the increment sequence. Any
increment sequence is fine as long as h1 = 1 and some other choices are
better than others.
• Shell sort makes multiple passes through a list and sorts a number of equally
sized sets using the insertion sort.
• Shell sort improves on the efficiency of insertion sort by quickly shifting
values to their destination.

Shell sort
• Shell sort is also known as diminishing increment sort.
• The distance between comparisons decreases as the sorting algorithm runs until
the last phase in which adjacent elements are compared
hk, for every i, we have a[ i ] ≤ a [ i + hk ]
• After each phase and some increment
all elements spaced hk apart are sorted.
• The file is said to be hk – sorted.

Empirical Analysis of Shell sort
Advantage:
• Advantage of Shellsort is that its only efficient for medium size lists. For bigger
lists, the algorithm is not the best choice. Fastest of all O(N^2) sorting
algorithms.
• 5 times faster than the bubble sort and a little over twice as fast as the insertion
sort, its closest competitor.

Empirical Analysis of Shell sort
Disadvantage:
• Disadvantage of Shellsort is that it is a complex algorithm and its not nearly as
efficient as the merge, heap, and quick sorts.
• The shell sort is still significantly slower than the merge, heap, and quick sorts,
but its relatively simple algorithm makes it a good choice for sorting lists of less
than 5000 items unless speed important. It's also an excellent choice for
repetitive sorting of smaller lists.

Shell sort Best Case
• Best Case: The best case in the shell sort is when the array is already sorted
in the right order. The number of comparisons is less.

Shell sort Worst Case
• The running time of Shellsort depends on the choice of increment sequence.
• The problem with Shell’s increments is that pairs of increments are not
necessarily relatively prime and smaller increments can have little effect.

Shell sort Example
Sort: 18 32 12 5 38 33 16 2
8 Numbers to be sorted, Shell’s increment will be floor(n/2)
* floor(8/2)  floor(4) = 4
1 2 3
18 32 12 5 38 33 16 2
increment 4: 4 (visualize underlining)
Step 1) Only look at 18 and 38 and sort in order ;
18 and 38 stays at its current position because they are in order.

Shell sort Example
Sort: 18 32 12 5 38 33 16 2
8 Numbers to be sorted, Shell’s increment will be floor(n/2)
* floor(8/2)  floor(4) = 4
1 2 3
18 32 12 5 38 33 16 2
increment 4: 4 (visualize underlining)
2 and 5 need to be switched to be in order.

Shell sort Example (contd..)
Sort: 18 32 12 5 38 33 16 2
* floor(2/2)  floor(1) = 1
increment 1: 1
12 2 16 5 18 32 38 33
2 5 12 16 18 32 33 38
The last increment or phase of Shellsort is basically an Insertion
Sort algorithm.

Divide and Conquer Problem
• In divide and conquer approach, a problem is divided into smaller problems,
then the smaller problems are solved independently, and finally the solutions of
smaller problems are combined into a solution for the large problem.
• Generally, divide-and-conquer algorithms have three parts −
• Divide the problem into a number of sub-problems that are smaller
instances of the same problem.
• Conquer the sub-problems by solving them recursively. If they are small
enough, solve the sub-problems as base cases.
• Combine the solutions to the sub-problems into the solution for the original
problem.
• Application of Divide and Conquer Approach
Finding the maximum and minimum of a sequence of numbers
Strassen’s matrix multiplication
Merge sort
Binary search
Quick Sort

Quick Sort
• Fastest known sorting algorithm in practice
• Average case: O(N log N) (we don’t prove it)
• Worst case: O(N2)
• But, the worst case seldom happens.
• Another divide-and-conquer recursive algorithm, like merge sort

Quick Sort
• Follows the divide-and-conquer paradigm.
• Divide: Partition (separate) the array A[p..r] into two (possibly empty) sub arrays
A[p..q–1] and A[q+1..r].
• Each element in A[p..q–1] < A[q].
• A[q] < each element in A[q+1..r].
• Index q is computed as part of the partitioning procedure.
• Conquer: Sort the two sub arrays by recursive calls to quick sort.
• Combine: The sub arrays are sorted in place – no work is needed to combine
them.
• How do the divide and combine steps of quick sort compare with those of merge
sort?

Pseudocode
Quicksort(A, p, r)
if p < r then
q := Partition(A, p, r);
Quicksort(A, p, q – 1);
Quicksort(A, q + 1, r)
Partition(A, p, r)
x, i := A[r], p – 1;
for j := p to r – 1 do
if A[j]  x then
i := i + 1;
A[i]  A[j]
A[i + 1]  A[r];
return i + 1
5
A[p..r]
Partition
A[p..q – 1] A[q+1..r]
5
 5  5

Example
p
2 5 8 3 9 4 1 7 10 6
r
initially: note: pivot (x) = 6
i j
next iteration: 2 5 8 3 9 4 1 7 10 6
8 3 9 4 1 7 10 6
8 3 9 4 1 7 10 6
3
j
8 9 4 1 7 10 6
i j
next iteration: 2 5
i j
next iteration: 2 5
i j
next iteration: 2 5
i
Partition(A, p, r)
x, i := A[r], p – 1;
if A[j]  x then
i := i + 1;
A[i]  A[j]
A[i + 1]  A[r];
return i + 1

Example (contd…)
next iteration: 2 5 3 8 9 4 1 7 10 6
i j
next iteration: 2 5 3 8 9 4 1 7 10 6
i j
next iteration: 2 5 3 4 9 8 1 7 10 6 Partition(A, p, r)
i j x, i := A[r], p – 1;
next iteration: 2 5 3
i
4 1 8
j
9 7 10 6 for j := p to r – 1 do
if A[j]  x then
next iteration:
next iteration:
2
2
5
5
3
i
3
4
4
1
1
8
8
9
j
9
7
7
10
10
6
6
i := i + 1;
A[i]  A[j]
A[i + 1]  A[r];
i j return i + 1
after final swap: 2 5 3 4 1 6 9 7 10 8
i j

Partitioning
•
• Select the last element A[r] in the subarray A[p..r] as the pivot – the element
around which to partition.
As the procedure executes, the array is partitioned into four (possibly empty)
regions.
1. A[p..i ] — All entries in this region are < pivot.
2. A[i+1..j – 1] — All entries in this region are > pivot.
3. A[r] = pivot.
4. A[j..r – 1] — Not known how they compare to pivot.
• The above hold before each iteration of the for loop, and constitute a loop
invariant. (4 is not part of the loopi.)

Correctness of Partition
• Use loop invariant.
• Initialization:
• Before first iteration
• A[p..i] and A[i+1..j – 1] are empty – Conds. 1 and 2 are satisfied
(trivially).
• r is the index of the pivot
• Cond. 3 is satisfied.
• Maintenance:
• Case 1: A[j] > x
• Increment j only.
• Loop Invariant is maintained.
Partition(A, p, r)
x, i := A[r], p – 1;
if A[j]  x then
i := i + 1;
A[i]  A[j]
A[i + 1]  A[r];
return i + 1

Case 1:
>x x
p i j r
 x > x
x
p i j r
 x > x

• Case 2: A[j]  x
• Increment i
• Swap A[i] and A[j]
• Condition 1 is maintained.
• Increment j
• Condition 2 is maintained.
• A[r] is unaltered.
• Condition 3 is
maintained.
 x x
p i j r
 x > x
 x > x
x
p i j r

• Termination:
• When the loop terminates, j = r, so all elements in A are
partitioned into one of the three cases:
• A[p..i]  pivot
• A[i+1..j – 1] > pivot
• A[r] = pivot
• The last two lines swap A[i+1] and A[r].
• Pivot moves from the end of the array to between the
two subarrays.
• Thus, procedure partition correctly performs the divide
step.

Complexity of Partition
• PartitionTime(n) is given by the number of
iterations in the for loop.
• (n) : n = r – p + 1.
Partition(A, p, r)
x, i := A[r], p – 1;
if A[j]  x then
i := i + 1;
A[i]  A[j]
A[i + 1]  A[r];
return i + 1

Partitioning in Quick sort
• A key step in the Quick sort algorithm is
partitioning the array
• We choose some (any) number p in the array to use as
a pivot
• We partition the array into three parts:
p
numbers less
than p
numbers greater than or
equal to p
p

Analysis of quick sort—best case
• Suppose each partition operation divides the array almost exactly in half
• Then the depth of the recursion in log2n
• Because that’s how many times we can halve n
• We note that
• Each partition is linear over its subarray
• All the partitions at one level cover the array

Partitioning at various levels

Best Case Analysis
• We cut the array size in half each time
• So the depth of the recursion in log2n
• At each level of the recursion, all the partitions at that level do work that is linear
in n
• O(log2n) * O(n) = O(n log2n)
• Hence in the best case, quicksort has time complexity O(n log2n)
• What about the worst case?

Worst case
• In the worst case, partitioning always divides the size n array into these three
parts:
• A length one part, containing the pivot itself
• A length zero part, and
• A length n-1 part, containing everything else
• We don’t recur on the zero-length part
• Recurring on the length n-1 part requires (in the worst case) recurring to depth n-
1

Worst case for quick sort
• In the worst case, recursion may be n levels deep (for an array of size n)
• But the partitioning work done at each level is still n
• O(n) * O(n) = O(n2)
• So worst case for Quicksort is O(n2)
• When does this happen?
• There are many arrangements that could make this happen
• Here are two common cases:
• When the array is already sorted
• When the array is inversely sorted (sorted in the opposite order)

Typical case for quick sort
• If the array is sorted to begin with, Quick sort is terrible: O(n2)
• It is possible to construct other bad cases
• However, Quick sort is usually O(n log2n)
• The constants are so good that Quick sort is generally the faster algorithm.
• Most real-world sorting is done by Quick sort

Picking a better pivot
• Before, we picked the first element of the sub array to use as a pivot
• If the array is already sorted, this results in O(n2) behavior
• It’s no better if we pick the last element
• We could do an optimal quick sort (guaranteed O(n log n)) if we always picked a
pivot value that exactly cuts the array in half
• Such a value is called a median: half of the values in the array are larger,
half are smaller
• The easiest way to find the median is to sort the array and pick the value in
the middle (!)

Quicksort for Small Arrays
• For very small arrays (N<= 20), quicksort does not perform as well as insertion
sort
• A good cutoff range is N=10
• Switching to insertion sort for small arrays can save about 15% in the running
time

Heap Sort
• Heap Sort is one of the best sorting methods being in-place and with no quadratic
worst-case running time. Heap sort involves building a Heap data structure from
the given array and then utilizing the Heap to sort the array.
• Def: A heap is a nearly complete binary tree with the following two properties:
• Structural property: all levels are full, except possibly the last one, which
is filled from left to right
• Order (heap) property: for any node x
Parent(x) ≥ x

Array Representation of Heaps
• A heap can be stored as an array A.
• Root of tree is A[1]
• Left child of A[i] = A[2i]
• Right child of A[i] = A[2i + 1]
• Parent of A[i] = A[ i/2 ]
• Heapsize[A] ≤ length[A]
• The elements in the subarray
A[(n/2+1) .. n] are leaves
119

Heap Types
• Max-heaps (largest element at root), have the max-heap property:
• for all nodes i, excluding the root:
• A[PARENT(i)] ≥ A[i]
• Min-heaps (smallest element at root), have the min-heap property:
• for all nodes i, excluding the root:
• A[PARENT(i)] ≤ A[i]

Adding/Deleting Nodes
• New nodes are always inserted at the bottom level (left to right)
• Nodes are removed from the bottom level (right to left)

Operations on Heaps
• Maintain/Restore the max-heap property
• MAX-HEAPIFY
• Create a max-heap from an unordered array
• BUILD-MAX-HEAP
• Sort an array in place
• HEAPSORT
• Priority queues

Maintaining the Heap Property
• Suppose a node is smaller than a child
• Left and Right subtrees of i are max-heaps
• To eliminate the violation:
• Exchange with larger child
• Move down the tree
• Continue until node is not smaller than children

Example
MAX-HEAPIFY(A, 2, 10)
A[2] violates the heap property
A[2]  A[4]
A[4] violates the heap property
A[4]  A[9]
Heap property restored

Maintaining the Heap Property
• Assumptions:
• Left and Right subtrees
of i are max-heaps
• A[i] may be smaller
than its children
Alg: MAX-HEAPIFY(A, i, n)
1. l ← LEFT(i)
2. r ← RIGHT(i)
3. if l ≤ n and A[l] > A[i]
4. then largest ←l
5. else largest ←i
6. if r ≤ n and A[r] > A[largest]
7. then largest ←r
8. if largest  i
9. then exchange A[i] ↔ A[largest]
10. MAX-HEAPIFY(A, largest, n)

MAX-HEAPIFY Running Time
• Intuitively:
• Running time of MAX-HEAPIFY is O(lg n)
• Can be written in terms of the height of the heap, as being O(h)
• Since the height of the heap is lg n
h
2h
O(h)

Building a Heap
• Convert an array A[1 … n] into a max-heap (n = length[A])
• The elements in the subarray A[(n/2+1) .. n] are leaves
• Apply MAX-HEAPIFY on elements between 1 and n/2
Alg: BUILD-MAX-HEAP(A)
1. n = length[A]
2. for i ← n/2 downto 1
3. do MAX-HEAPIFY(A, i, n)
2
1
4
2
1
3
3
4 5 6
16 9
7
10
8
14
9 10
8 7
4 1 3 2 16 9 10 14 8 7
A:

A 4 1 3 2 16 9 10 14 8 7
2
1
3
3
2
1
3
10
2
1
3
3
1 1 1
4 4 4
2
1
3
3
5 6
16 9
7
10
4 5 6 7 4 5 6 7 4
2 9
16
10 9 10 8
2 9
16
10 9 10 8 14
8 9 10
14 8 7 14 8 7 2 8 7
Example:
i = 5
2
16
3
10
1
16
2
14
3
10
5 6
7 9
7
3
4 5 6 7 4 5 6 7 4
14 9
16
10 9 3 8
14 9
7
10 9 3 8 8
8 9 10
2 8 7 2 8 1 2 4 1
i = 2
1
4
i = 1
1
4
i = 4 i = 3

Running Time of BUILD MAX HEAP
 Running time: O(n lg n)
• This is not an asymptotically tight upper bound
Alg: BUILD-MAX-HEAP(A)
1. n = length[A]
2. for i ← n/2 downto 1
3. do MAX-HEAPIFY(A, i, n)
O(lgn)
O(n)

• HEAPIFY takes O(h)  the cost of HEAPIFY on a node i is proportional to the height
of the node i in the tree h
 T(n)  ni hi
i0
h
 2i
hi
i0
Height Level
h0 = 3 (lgn)
h1 = 2
h2 = 1
h3 = 0
i = 0
i = 1
i = 2
i = 3 (lgn)
No. of nodes
20
21
22
23
hi = h – i
ni = 2i
130
height of the heap rooted at level i
number of nodes at level i

h
T(n)  ni hi
i0
h
Cost of HEAPIFY at level i  number of nodes at that level
 2i
hi Replace the values of ni and hi computed before
i0 2hi
h i
2h
i0
h
  Multiply by 2h both at the nominator and denominator and
write 2i as
1
2i

131
h
k0
 2h
k
k
2
Change variables: k = h - i
k0 2k

 n

k The sum above is smaller than the sum of all elements to 
and h = lgn
 O(n) The sum above is smaller than 2
Running time of BUILD-MAX-HEAP: T(n) = O(n)

Heapsort
• Goal:
• Sort an array using heap representations
• Idea:
• Build a max-heap from the array
• Swap the root (the maximum element) with the last element in the array
• “Discard” this last node by decreasing the heap size
• Call MAX-HEAPIFY on the new root
• Repeat this process until only one node remains
132

Example: A=[7, 4, 3, 1, 2]
133
MAX-HEAPIFY(A, 1, 3) MAX-HEAPIFY(A, 1, 2)

Algorithm: HEAPSORT(A)
1. BUILD-MAX-HEAP(A)
2. for i ← length[A] downto 2
3. do exchange A[1] ↔ A[i]
4. MAX-HEAPIFY(A, 1, i - 1)
• Running time: O(nlgn) --- Can be shown to be Θ(nlgn)
O(n)
O(lgn)
n-1 times
134

n
(sorted)
MERGE
Merge Sort
• Divide stage: Split the n-element
sequence into two subsequences of
n/2 elements each
• Conquer stage: Recursively sort the
two subsequences
• Combine stage: Merge the two
sorted subsequences into one sorted
sequence (the solution)
n
(unsorted)
n/2
(unsorted)
n/2
(unsorted)
MERGE SORT MERGE SORT
n/2
(sorted)
n/2
(sorted)

Merging Sorted Sequences
•Combines the sorted
subarrays A[p..q] and
A[q+1..r] into one sorted
array A[p..r]
•Makes use of two working
arrays L and R which
initially hold copies of the
two subarrays
•Makes use of sentinel
value () as last element
to simplify logic
(1)
(n)
(1)
(n)

Merge Sort Algorithm
(1)
T(n/2)
T(n/2)
(n)
T(n) = 2T(n/2) + (n)

Analysis of Merge Sort
Analysis of recursive calls …

Analysis of Merge Sort
T(n) = cn(lg n + 1)
= cnlg n + cn
T(n) is (n lg n)

Analysis of Different Sorting
Algorithms
Comparison based sorting:
• Bubble sort and Insertion sort
Average and worst case time complexity: n^2
Best case time complexity: n when array is already sorted.
Worst case: when the array is reverse sorted.
• Selection sort
Best, average and worst case time complexity: n^2 which is independent of
distribution of data.
• Merge sort
Best, average and worst case time complexity: nlogn which is independent of

Analysis of Different sorting
Algorithms Contd...
• Heap sort
Best, average and worst case time complexity: nlogn which is independent of
• Quick sort
It is a divide and conquer approach with recurrence relation: T(n) = T(k) + T(n-k-
1) + cn
• Worst case: when the array is sorted or reverse sorted, the partition algorithm
divides the array in two subarrays with 0 and n-1 elements. Therefore,
• T(n) = T(0) + T(n-1) + cn Solving this we get, T(n) = O(n^2)
• Best case and Average case: On an average, the partition algorithm divides the
array in two subarrays with equal size. Therefore,
• T(n) = 2T(n/2) + cn Solving this we get, T(n) = O(nlogn)

Here w
e will see some sorting methods. W
e will see few of them. Some sorting techniques are comparison
based sort, some are non-comparison based sorting technique.
Comparison Based Soring techniques are bubble sort, selection sort, insertion sort, Merge sort, quicksort,
heap sort etc. These techniques are considered as comparison based sort because in these techniques the
values are compared, and placed into sorted position in different phases. Here w
e will see time complexity of
these techniques.
Analysis
Type
Bubble
Sort
Selection
Sort
Insertion
Sort
Merge
Sort
Quick
Sort
Heap
Sort
Best Case
2
O(n )
2
O(n ) O(n) O(log n) O(log n) O(logn)
Average
Case
2
O(n )
2
O(n )
2
O(n ) O(log n) O(log n) O(log n)
Worst
Case
2)
O(n
2
O(n )
2
O(n ) O(log n)
2
O(n ) O(log n)
Summary of time complexity of
comparison-based Sorting Techniques

Analysis of Different Sorting
Algorithms :Non Comparison Based
Non-comparison based sorting :
In non-comparison based sorting, elements of array are not compared with each
other to find the sorted array.
• Radix sort
Best, average and worst case time complexity: nk where k is the maximum
number of digits in elements of array.
• Count sort
Best, average and worst case time complexity: n+k where k is the size of count
array.
• Bucket sort
Best and average time complexity: n+k where k is the number of buckets.
Worst case time complexity: n^2 if all elements belong to same bucket.

Summary of time complexity of non-
comparison based Sorting Techniques

Matrix Multiplication Problem
Divide and Conquer
Following is simple Divide and Conquer method to multiply two square matrices.
1) Divide matrices A and B in 4 sub-matrices of size N/2 x N/2 as shown in the below
diagram.
2) Calculate following values recursively. ae + bg, af + bh, ce + dg and cf + dh.

• Naive matrix multiplication
• Let us start with two square matrices A and B which are both of size n by n. In
the product C = A X B we define the entry cij, the entry in the ith row and the jth
column of A, as being the dot product of the ith row of A with the jth column of
B. Taking the dot product of vectors just means that you take the products of the
individual components and then add up the results.
• Complexity =O(n^3)

• In the above method, we do 8 multiplications for matrices of size N/2 x N/2 and
4 additions. Addition of two matrices takes O(N2) time. So the time complexity
can be written
• T(N) = 8T(N/2) + O(N2)
• From Master's Theorem, time complexity of above method is O(N3) which is
unfortunately same as the above naive method.
• Simple Divide and Conquer also leads to O(N3), can there be a better way?
In the above divide and conquer method, the main component for high time
complexity is 8 recursive calls. The idea of Strassen’s method is to reduce the
number of recursive calls to 7. Strassen’s method is similar to above simple
divide and conquer method in the sense that this method also divide matrices to
sub-matrices of size N/2 x N/2 as shown in the above diagram, but in Strassen’s
method, the four sub-matrices of result are calculated using following formulae.

Time Complexity of Strassen’s Method
• Addition and Subtraction of two matrices takes O(N2) time. So time complexity
can be written as
T(N) = 7T(N/2) + O(N2)
• From Master's Theorem, time complexity of above method is O(NLog7) which is
approximately O(N2.8074)
• Generally Strassen’s Method is not preferred for practical applications for
following reasons.
1) The constants used in Strassen’s method are high and for a typical application
Naive method works better.
2) For Sparse matrices, there are better methods especially designed for them.
3) The submatrices in recursion take extra space.

Convex vs. Concave
• A polygon P is convex if for every pair of points x and y in P, the line xy is also in
P; otherwise, it is called concave.
y
P x y
P x
convex
concave
Convex Hull

153
The convex hull problem
• The convex hull of a set of planar points is the smallest convex polygon
containing all of the points.
concave polygon: convex polygon:

Graham’s Scan
• Graham's scan is a method of finding the convex hull of a finite set of points in
the plane with time complexity O(n log n).
• It is named after Ronald Graham, who published the original algorithm in 1972.
• The algorithm finds all vertices of the convex hull ordered along its boundary.
• Start at point guaranteed to be on the hull. (the point with the minimum y value)
• Sort remaining points by polar angles of vertices relative to the first point.
• Go through sorted points, keeping vertices of points that have left turns and
dropping points that have right turns.

Graham’s Runtime
• Graham’s scan is O(n log n) due to initial sort of
angles.

Convex Hull by Divide-and-Conquer
• First, sort all points by their x coordinate.
– ( O(n log n) time)
• Then divide and conquer:
– Find the convex hull of the left half of points.
– Find the convex hull of the right half of points.
– Merge the two hulls into one.

5
Hull of S by exhaustive search;

Design & Analysis of Algorithm course .pptx

Design & Analysis of Algorithm course .pptx

More Related Content

Similar to Design & Analysis of Algorithm course .pptx (20)

Recently uploaded (20)

Design & Analysis of Algorithm course .pptx