01 - Analysis of Efficiency.pptx

Topic 1
Fundamentals of the Analysis of
Algorithm Efficiency

2-1
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2 1-1
What is an algorithm?
An algorithm is a sequence of unambiguous instructions
for solving a problem, i.e., for obtaining a required
output for any legitimate input in a finite amount of
time.
“computer”
problem
algorithm
input output

2-2
Algorithm
• Representable in various forms
• Unambiguity/clearness
• Effectiveness
• Finiteness/termination
• Correctness

2-3
Historical Perspective
 Euclid’s algorithm for finding the greatest common divisor
 Muhammad ibn Musa al-Khwarizmi – 9th century
mathematician
www.lib.virginia.edu/science/parshall/khwariz.html

2-4
Euclid’s Algorithm
Problem: Find gcd(m,n), the greatest common divisor of two
nonnegative, not both zero integers m and n
Examples: gcd(60,24) = 12, gcd(60,0) = 60,
Euclid’s algorithm is based on repeated application of equality
gcd(m,n) = gcd(n, m mod n)
until the second number becomes 0, which makes the problem
trivial.
Example: gcd(60,24) = gcd(24,12) = gcd(12,0) = 12
gcd(0,0) = ?

2-5
Two descriptions of Euclid’s algorithm
Step 1 If n = 0, return m and stop; otherwise go to Step 2
Step 2 Divide m by n and assign the value of the remainder to r
Step 3 Assign the value of n to m and the value of r to n. Go to
Step 1.
while n ≠ 0 do
r ← m mod n
m← n
n ← r
return m

2-6
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2
Analysis of algorithms
 Issues:
• correctness
• time efficiency (how fast?)
• space efficiency (how much memory required?)
• optimality
 Approaches:
• theoretical analysis
• empirical analysis

2-7
Theoretical analysis of time efficiency
 Time efficiency is a function of input size.
 It is analyzed by determining the number of repetitions of the
basic operation of the algorithm.
 Basic operation: the operation that contributes the most towards
the running time of the algorithm
T(n) ≈ copC(n)
running time execution time
for basic operation
or cost
Number of times
basic operation is
executed
input size Note: Different basic
operations may cost
differently!

2-8
Input size and basic operation examples
Problem Input size measure Basic operation
Searching for key in a
list of n items
Number of list’s items, i.e.
n
Key comparison
Multiplication of two
matrices
Matrix dimensions or total
number of elements
Multiplication of two
numbers
Checking primality of a
given integer n
n’size = number of digits
(in binary representation)
Division
Typical graph problem #vertices and/or edges
Visiting a vertex or
traversing an edge

2-9
Best-case, average-case, worst-case
For some algorithms, the efficiency (running time) can be different
for different inputs of the same size:
 Worst case: Cworst(n) – maximum over inputs of size n
 Best case: Cbest(n) – minimum over inputs of size n
 Average case: Cavg(n) – “average” over inputs of size n
• Average refers to the number of operations required for a typical input, on
average.
• NOT the average of worst and best case

2-10
Example: Sequential search
 Worst case
 Best case
 Average case
n key comparisons
1 comparisons
(n+1)/2, assuming K is in A

2-11
Types of formulas for basic operation’s count
 Exact formula
e.g., C(n) = n(n-1)/2
 Formula indicating order of growth with specific
multiplicative constant
e.g., C(n) ≈ 0.5 n2
 Formula indicating order of growth with unknown
multiplicative constant
e.g., C(n) ≈ cn2

2-12
Order of growth
 How slow/fast the algorithm will be if the input size is
changed?
 Order of growth is defined within a constant multiple as
n→∞
• For small values of n, the efficiency does not really matter.
 Example:
• How much faster will algorithm run on computer that is
twice as fast?
• How much longer does it take to solve problem of double
input size?

2-13
Values of some important functions as n  

2-14
Asymptotic order of growth
A way of comparing functions that ignores constant factors and
small input sizes (because?)
 t(n) is O(g(n)):
class of functions t(n) that grow no faster than g(n)
 t(n) is Θ(g(n)):
class of functions t(n) that grow at same rate as g(n)
 f(n) is Ω(g(n)):
class of functions t(n) that grow at least as fast as g(n)

2-15
O-notation
Definition: t(n) is in O(g(n)), denoted t(n)  O(g(n)), if order of
growth of t(n) ≤ order of growth of g(n) (within constant
multiple), i.e., there exist positive constant c and non-negative
integer n0 such that t(n) ≤ c g(n) for every n ≥ n0
Examples:
 10n is in O(n2)
 5n+20 is in O(n)

2-16
-notation
Definition: A function t(n) is said to be in (g(n)), denoted t(n) 
(g(n)), if t(n) is bounded below by some constant multiple of g(n)
for all large n, i.e., if there exist some positive constant c and some
nonnegative integer n0 such that
t(n)  cg(n) for all n  n0
Exercises: prove the following
• 10n2  (n2)
• 0.3n2 - 2n  (n2)
• 0.1n3  (n2)

2-17
-notation
Definition
A function t(n) is said to be in (g(n)), denoted t(n)  (g(n)), if t(n)
is bounded both above and below by some positive constant
multiples of g(n) for all large n, i.e., if there exist some positive
constant c1 and c2 and some nonnegative integer n0 such that
c2 g(n)  t(n)  c1 g(n) for all n  n0
Exercises: prove the following
• 10n2  (n2)
• 0.3n2 - 2n  (n2)
• (1/2)n(n+1)  (n2)

2-18
(g(n)), functions that grow at least as fast as g(n)
(g(n)), functions that grow at the same rate as g(n)
O(g(n)), functions that grow no faster than g(n)
g(n)
>=
<=
=

2-20
Some properties of asymptotic order of growth
 f(n)  O(f(n))
 f(n)  O(g(n)) iff g(n) (f(n))
 If f (n)  O(g (n)) and g(n)  O(h(n)) , then f(n)  O(h(n))
Note similarity with a ≤ b ≤ c
 If f1(n)  O(g1(n)) and f2(n)  O(g2(n)) , then
f1(n) + f2(n)  O(max{g1(n), g2(n)})
Also, 1in (f(i)) =  (1in f(i))
Exercise: Can you prove
these properties?

2-21
Establishing order of growth using limits
lim T(n)/g(n) =
0 order of growth of T(n) < order of growth of g(n)
c > 0 order of growth of T(n) = order of growth of g(n)
∞ order of growth of T(n) > order of growth of g(n)
Examples:
• 10n vs. n2
• n(n+1)/2 vs. n2
n→∞

2-23
Orders of growth of some important functions
 All logarithmic functions loga n belong to the same class
(log n) no matter what the logarithm’s base a > 1 is
because
 All polynomials of the same degree k belong to the same class:
aknk + ak-1nk-1 + … + a0  (nk)
 Exponential functions an have different orders of growth for
different a’s
 order log n < order n (>0) < order an < order n! < order nn
log𝑎 𝑛 = log𝑏 𝑛 / log𝑏 𝑎

2-24
Basic asymptotic efficiency classes
1 constant
log n logarithmic
n linear
n log n n-log-n
n2 quadratic
n3 cubic
2n exponential
n! factorial

2-25
Basic asymptotic efficiency classes

2-26
Time efficiency of nonrecursive algorithms
General Plan for Analysis
 Decide on parameter n indicating input size
 Identify algorithm’s basic operation
 Determine worst, average, and best cases for input of size n
 Set up a sum for the number of times the basic operation is
executed
 Simplify the sum using standard formulas and rules (see
Appendix A)

2-27
Useful summation formulas and rules
lin1 = 1+1+…+1 = n - l + 1
In particular, lin1 = n - 1 + 1 = n  (n)
1in i = 1+2+…+n = n(n+1)/2  n2/2  (n2)
1in i2 = 12+22+…+n2 = n(n+1)(2n+1)/6  n3/3  (n3)
0in ai = 1 + a +…+ an = (an+1 - 1)/(a - 1) for any a  1
In particular, 0in 2i = 20 + 21 +…+ 2n = 2n+1 - 1  (2n )
(ai ± bi ) = ai ± bi ; cai = cai ; liuai = limai + m+1iuai
1𝑘
+ 2𝑘
+ 3𝑘
+ ⋯ + 𝑛𝑘
≤ 𝑛𝑘
+ 𝑛𝑘
+ 𝑛𝑘
+ ⋯ + 𝑛𝑘
= 𝑛𝑘+1
∈ Θ(𝑛𝑘+1
)

2-28
Example 1: Maximum element
C(n) = 1in-1 1 = n-1 = (n) comparisons

2-29
Example 2: Element uniqueness problem
𝐶 𝑛 =
𝑖=0
𝑛−2
𝑗=𝑖+1
𝑛−1
1 =
𝑖=0
𝑛−2
𝑛 − 𝑖 − 1 = 𝑛 − 1 + 𝑛 − 2 + ⋯ + 1
= 𝑛(𝑛 − 1)/2 = 𝜃(𝑛2
)

2-30
Example 3: Matrix multiplication
𝐶 𝑛 =
𝑖=0
𝑛−1
𝑗=0
𝑛−1
𝑘=0
𝑛−1
1 =
𝑖=0
𝑛−1
𝑗=0
𝑛−1
𝑛 =
𝑖=0
𝑛−1
𝑛2
= 𝑛3
multiplications
also 𝑛3 additions

2-31
Example 4: Counting binary digits
It cannot be investigated the way the previous examples are.
The halving game: Find integer i such that n/ ≤ 1.
Answer: i ≥ log n. So, T(n) = (log n) divisions.
Another solution: Using recurrence relations.
2𝑖

2-32
Plan for Analysis of Recursive Algorithms
 Decide on a parameter indicating an input’s size.
 Identify the algorithm’s basic operation.
 Check whether the number of times the basic op. is executed may
vary on different inputs of the same size. (If it may, the worst,
average, and best cases must be investigated separately.)
 Set up a recurrence relation with an appropriate initial condition
expressing the number of times the basic op. is executed.
 Solve the recurrence (or, at the very least, establish its solution’s
order of growth) by backward substitutions or another method.

2-33
Example 1: Recursive evaluation of n!
Definition: n ! = 1  2  … (n-1)  n for n ≥ 1 and 0! = 1
Recursive definition of n!: F(n) = F(n-1)  n for n ≥ 1 and
F(0) = 1
Size:
Basic operation:
Recurrence relation:
n
multiplication
M(n) = M(n-1) + 1
M(0) = 0

2-34
Solving the recurrence for M(n)
M(n) = M(n-1) + 1, M(0) = 0
M(n) = M(n-1) + 1
= (M(n-2) + 1) + 1 = M(n-2) + 2
= (M(n-3) + 1) + 2 = M(n-3) + 3
…
= M(n-i) + i
= M(0) + n
= n
The method is called backward substitution.
M(n) = M(n-1)+1
M(n-1) = M(n-2)+1
M(n-2) = M(n-3)+1
M(n-3) = M(n-4)+1
…
We can get rid of M(n-i) if we set i=n,
because we know the value of M(0).

2-35
Example 2: The Tower of Hanoi Puzzle
1
2
3
Recurrence for number of moves:
M(n) = M(n-1) + 1+ M(n-1)

2-36
Solving the recurrence for M(n)
M(n) = 2M(n-1) + 1, M(0) = 1
M(n) = 2M(n –1) + 1
= 2[2M(n–2) + 1]+ 1 = 22M(n–2)+2+1
= 22[2M(n–3) + 1]+2+1 = 23M(n–3)+22+2+1
= 23[2M(n–4)+1]+22+2+1 = 24M(n–4)+23+22+2+1
…
= 2iM(n–i)+2i–1+ 2i–2+ 2i–3+… 23+22+2+1
= 2nM(0)+ 2n–1+ 2n–2+ 2n–3+… 23+22+2+1
= 2n+ 2n–1+ 2n–2+ 2n–3+… 23+22+2+1 = 2n+1 – 1
M(n) = 2M(n-1)+1
M(n-1) = 2M(n-2)+1
M(n-2) = 2M(n-3)+1
M(n-3) = 2M(n-4)+1
…
We can get rid of M(n-i) if
we set i=n, because we
know the value of M(0).

2-37
Example 3: Counting #bits
A( ) = A( ) + 1, A( ) = 0 (using the Smoothness Rule)
= (A( ) + 1) + 1 = A( ) + 2
= A( ) + i
= A( ) + k = k + 0
=
2𝑘
2𝑘−1 20
2𝑘−2
log2 𝑛
2𝑘−2
2𝑘−𝑖
2𝑘−𝑘
A(n) = A( ) + 1, A(1) = 0
𝑛/2
Assume n = 2k
i.e., k = log2 n

2-38
Smoothness Rule
 Let f(n) be a nonnegative function defined on the set of
natural numbers. f(n) is called smooth if it is eventually
nondecreasing and
f(2n) ∈ Θ (f(n))
• Functions that do not grow too fast, including logn, n, nlogn,
and n where >=0 are smooth.
 Smoothness rule
Let T(n) be an eventually nondecreasing function and f(n) be
a smooth function. If
T(n) ∈ Θ (f(n)) for values of n that are powers of b,
where b>=2, then
T(n) ∈ Θ (f(n)) for any n.

2-39
Empirical analysis of time efficiency
 Many algorithms cannot be analyzed mathematically.
The alternative way is the Empirical Analysis.
 The steps are:
• Select a specific (typical) sample of inputs
• Decide on the efficiency metric (execution time or operation
count)
• Implement the algorithm
• Run the code and record the observed data (time or number
of operations)
• Analyze the data

2-40
Empirical analysis of time efficiency
 The recorded time is usually not accurate. In general,
we repeat the experiment many times and take
average (or median) value.
 Empirical analysis is usually done on several inputs of
different sizes.
 By measuring the time at inputs of size N, 2N, 3N, …
and calculating their ratios, we can guess the
efficiency class of the algorithm (linear, log, square
etc).

2-41
Algorithm Visualization
 We have studied mathematical and empirical analysis
of algorithms. A third way is Algorithm Visualization.
 It uses static images or animations to show the
performance of algorithm.
Initial and final states
of a sorting algorithm
using bar chart
Initial and final states
of a sorting algorithm
using scatter plot

2-42
Summary
 There are two kinds of algorithm efficiency: time efficiency and space
efficiency. Time efficiency indicates how fast the algorithm runs; space
efficiency deals with the extra space it requires.
 An algorithm’s time efficiency is principally measured as a function of its
input size by counting the number of times its basic operation is executed. A
basic operation is the operation that contributes the most to running time.
Typically, it is the most time-consuming operation in the algorithm’s
innermost loop.
 For some algorithms, the running time can differ considerably for inputs of
the same size, leading to worst-case efficiency, average-case efficiency, and
best-case efficiency.
 The established framework for analyzing time efficiency is primarily
grounded in the order of growth of the algorithm’s running time as its input
size goes to infinity.
 The notations O, Omega, and Theta are used to indicate and compare the
asymptotic orders of growth of functions expressing algorithm efficiencies.
 The efficiencies of a large number of algorithms fall into the following few
classes: constant, logarithmic, linear, linearithmic, quadratic, cubic, and
exponential.

2-43
Summary
 The main tool for analyzing the time efficiency of a nonrecursive algorithm is
to set up a sum expressing the number of executions of its basic operation
and ascertain the sum’s order of growth.
 The main tool for analyzing the time efficiency of a recursive algorithm is to
set up a recurrence relation expressing the number of executions of its basic
operation and ascertain the solution’s order of growth.
 Empirical analysis of an algorithm is performed by running a program
implementing the algorithm on a sample of inputs and analyzing the data
observed (the basic operation’s count or physical running time). This often
involves generating pseudorandom numbers. The applicability to any
algorithm is the principal strength of this approach; the dependence of
results on the particular computer and instance sample is its main weakness.
 Algorithm visualization is the use of images to convey useful information
about algorithms. The two principal variations of algorithm visualization are
static algorithm visualization and dynamic algorithm visualization (also called
algorithm animation).

01 - Analysis of Efficiency.pptx

More Related Content

Similar to 01 - Analysis of Efficiency.pptx (20)

Recently uploaded (20)

01 - Analysis of Efficiency.pptx