2. 2
Recurrences and Running Time
• An equation or inequality that describes a function in
terms of its value on smaller inputs.
T(n) = T(n-1) + n
• Recurrences arise when an algorithm contains recursive
calls to itself
• What is the actual running time of the algorithm?
• Need to solve the recurrence
– Find an explicit formula of the expression
– Bound the recurrence by an expression that involves n
3. 3
Example Recurrences
• T(n) = T(n-1) + n Θ(n2
)
– Recursive algorithm that loops through the input to
eliminate one item
• T(n) = T(n/2) + c Θ(lgn)
– Recursive algorithm that halves the input in one step
• T(n) = T(n/2) + n Θ(n)
– Recursive algorithm that halves the input but must
examine every item in the input
• T(n) = 2T(n/2) + 1 Θ(n)
– Recursive algorithm that splits the input into 2 halves
and does a constant amount of other work
4. 4
Recurrent Algorithms
BINARY-SEARCH
• for an ordered array A, finds if x is in the array A[lo…hi]
Alg.: BINARY-SEARCH (A, lo, hi, x)
if (lo > hi)
return FALSE
mid (lo+hi)/2
if x = A[mid]
return TRUE
if ( x < A[mid] )
BINARY-SEARCH (A, lo, mid-1, x)
if ( x > A[mid] )
BINARY-SEARCH (A, mid+1, hi, x)
12
11
10
9
7
5
3
2
1 2 3 4 5 6 7 8
mid
lo hi
5. 5
Example
• A[8] = {1, 2, 3, 4, 5, 7, 9, 11}
– lo = 1 hi = 8 x = 7
mid = 4, lo = 5, hi = 8
mid = 6, A[mid] = x
Found!
11
9
7
5
4
3
2
1
11
9
7
5
4
3
2
1
1 2 3 4 5 6 7 8
8
7
6
5
6. 6
Another Example
• A[8] = {1, 2, 3, 4, 5, 7, 9, 11}
– lo = 1 hi = 8 x = 6
mid = 4, lo = 5, hi = 8
mid = 6, A[6] = 7, lo = 5, hi = 5
11
9
7
5
4
3
2
1
11
9
7
5
4
3
2
1
1 2 3 4 5 6 7 8
11
9
7
5
4
3
2
1 mid = 5, A[5] = 5, lo = 6, hi = 5
NOT FOUND!
11
9
7
5
4
3
2
1
low high
low
low
high
high
7. 7
Analysis of BINARY-SEARCH
Alg.: BINARY-SEARCH (A, lo, hi, x)
if (lo > hi)
return FALSE
mid (lo+hi)/2
if x = A[mid]
return TRUE
if ( x < A[mid] )
BINARY-SEARCH (A, lo, mid-1, x)
if ( x > A[mid] )
BINARY-SEARCH (A, mid+1, hi, x)
• T(n) = c +
– T(n) – running time for an array of size n
constant time: c2
same problem of size n/2
same problem of size n/2
constant time: c1
constant time: c3
T(n/2)
8. 8
Methods for Solving Recurrences
• Iteration method
• Substitution method
• Recursion tree method
• Master method
9. 9
The Iteration Method
• Convert the recurrence into a summation and try
to bound it using known series
– Iterate the recurrence until the initial condition is
reached.
– Use back-substitution to express the recurrence in
terms of n and the initial (boundary) condition.
10. 10
The Iteration Method
T(n) = c + T(n/2)
T(n) = c + T(n/2)
= c + c + T(n/4)
= c + c + c + T(n/8)
Assume n = 2k
T(n) = c + c + … + c + T(1)
= clgn + T(1)
= Θ(lgn)
k times
T(n/2) = c + T(n/4)
T(n/4) = c + T(n/8)
11. 11
Iteration Method – Example
T(n) = n + 2T(n/2)
T(n) = n + 2T(n/2)
= n + 2(n/2 + 2T(n/4))
= n + n + 4T(n/4)
= n + n + 4(n/4 + 2T(n/8))
= n + n + n + 8T(n/8)
… = in + 2i
T(n/2i
)
= kn + 2k
T(1)
= nlgn + nT(1) = Θ(nlgn)
Assume: n = 2k
T(n/2) = n/2 + 2T(n/4)
13. 13
Substitution method
• Guess a solution
– T(n) = O(g(n))
– Induction goal: apply the definition of the asymptotic notation
• T(n) ≤ d g(n), for some d > 0 and n ≥ n0
– Induction hypothesis: T(k) ≤ d g(k) for all k < n
• Prove the induction goal
– Use the induction hypothesis to find some values of the
constants d and n0 for which the induction goal holds
(strong induction)
14. 14
Example: Binary Search
T(n) = c + T(n/2)
• Guess: T(n) = O(lgn)
– Induction goal: T(n) ≤ d lgn, for some d and n ≥ n0
– Induction hypothesis: T(n/2) ≤ d lg(n/2)
• Proof of induction goal:
T(n) = T(n/2) + c ≤ d lg(n/2) + c
= d lgn – d + c ≤ d lgn
if: – d + c ≤ 0, d ≥ c
• Base case?
15. 15
Example 2
T(n) = T(n-1) + n
• Guess: T(n) = O(n2
)
– Induction goal: T(n) ≤ c n2
, for some c and n ≥ n0
– Induction hypothesis: T(n-1) ≤ c(n-1)2
for all k < n
• Proof of induction goal:
T(n) = T(n-1) + n ≤ c (n-1)2
+ n
= cn2
– (2cn – c - n) ≤ cn2
if: 2cn – c – n ≥ 0 c ≥ n/(2n-1) c ≥ 1/(2 –
1/n)
– For n ≥ 1 2 – 1/n ≥ 1 any c ≥ 1 will work
16. 16
Example 3
T(n) = 2T(n/2) + n
• Guess: T(n) = O(nlgn)
– Induction goal: T(n) ≤ cn lgn, for some c and n ≥ n0
– Induction hypothesis: T(n/2) ≤ cn/2 lg(n/2)
• Proof of induction goal:
T(n) = 2T(n/2) + n ≤ 2c (n/2)lg(n/2) + n
= cn lgn – cn + n ≤ cn lgn
if: - cn + n ≤ 0 c ≥ 1
• Base case?
17. 17
Changing variables
– Rename: m = lgn n = 2m
T (2m
) = 2T(2m/2
) + m
– Rename: S(m) = T(2m
)
S(m) = 2S(m/2) + m S(m) = O(mlgm)
(demonstrated before)
T(n) = T(2m
) = S(m) = O(mlgm)=O(lgnlglgn)
Idea: transform the recurrence to one that you
have seen before
T(n) = 2T( ) + lgn
n
18. 18
The recursion-tree method
Convert the recurrence into a tree:
– Each node represents the cost incurred at various
levels of recursion
– Sum up the costs of all levels
Used to “guess” a solution for the recurrence
19. 19
Example 1
W(n) = 2W(n/2) + n2
• Subproblem size at level i is: n/2i
• Subproblem size hits 1 when 1 = n/2i
i = lgn
• Cost of the problem at level i = (n/2i
)2
No. of nodes at level i = 2i
• Total cost:
W(n) = O(n2
)
2
2
0
2
1
lg
0
2
lg
1
lg
0
2
2
)
(
2
1
1
1
)
(
2
1
2
1
)
1
(
2
2
)
( n
n
O
n
n
O
n
n
n
W
n
n
W
i
i
n
i
i
n
n
i
i
20. 20
Example 2
E.g.: T(n) = 3T(n/4) + cn2
• Subproblem size at level i is: n/4i
• Subproblem size hits 1 when 1 = n/4i
i = log4n
• Cost of a node at level i = c(n/4i
)2
• Number of nodes at level i = 3i
last level has 3log
4
n
= nlog
4
3
nodes
• Total cost:
T(n) = O(n2
)
)
(
16
3
1
1
16
3
16
3
)
( 2
3
log
2
3
log
2
0
3
log
2
1
log
0
4
4
4
4
n
O
n
cn
n
cn
n
cn
n
T
i
i
i
n
i
21. 21
Example 2 - Substitution
T(n) = 3T(n/4) + cn2
• Guess: T(n) = O(n2
)
– Induction goal: T(n) ≤ dn2
, for some d and n ≥ n0
– Induction hypothesis: T(n/4) ≤ d (n/4)2
• Proof of induction goal:
T(n) = 3T(n/4) + cn2
≤ 3d (n/4)2
+ cn2
= (3/16) d n2
+ cn2
≤ d n2
if: d ≥ (16/13)c
• Therefore: T(n) = O(n2
)
22. 22
Example 3 (simpler proof)
W(n) = W(n/3) + W(2n/3) + n
• The longest path from the root to a
leaf is: n
(2/3)n (2/3)2
n … 1
• Subproblem size hits 1 when 1 =
(2/3)i
n i=log3/2n
• Cost of the problem at level i = n
• Total cost:
W(n) = O(nlgn)
3/ 2
lg
( ) ... (log ) ( lg )
3
lg
2
n
W n n n n n n O n n
23. 23
Example 3
W(n) = W(n/3) + W(2n/3) + n
• The longest path from the root to a
leaf is: n
(2/3)n (2/3)2
n … 1
• Subproblem size hits 1 when 1 =
(2/3)i
n i=log3/2n
• Cost of the problem at level i = n
• Total cost:
W(n) = O(nlgn)
3/ 2
3/ 2
(log ) 1
(log )
0
( ) ... 2 (1)
n
n
i
W n n n n W
3/ 2
3/ 2
log
log 2
3/2
0
lg 1
1 log ( ) ( ) lg ( )
lg3/ 2 lg3/ 2
n
i
n
n n n n O n n O n n n O n
24. 24
Example 3 - Substitution
W(n) = W(n/3) + W(2n/3) + O(n)
• Guess: W(n) = O(nlgn)
– Induction goal: W(n) ≤ dnlgn, for some d and n ≥ n0
– Induction hypothesis: W(k) ≤ d klgk for any K < n
(n/3, 2n/3)
• Proof of induction goal:
Try it out as an exercise!!
• T(n) = O(nlgn)
25. 25
Master’s method
• “Cookbook” for solving recurrences of the form:
where, a ≥ 1, b > 1, and f(n) > 0
Idea: compare f(n) with nlog
b
a
• f(n) is asymptotically smaller or larger than nlog
b
a
by a
polynomial factor n
• f(n) is asymptotically equal with nlog
b
a
)
(
)
( n
f
b
n
aT
n
T
26. 26
Master’s method
• “Cookbook” for solving recurrences of the form:
where, a ≥ 1, b > 1, and f(n) > 0
Case 1: if f(n) = O(nlog
b
a -
) for some > 0, then: T(n) = (nlog
b
a
)
Case 2: if f(n) = (nlog
b
a
), then: T(n) = (nlog
b
a
lgn)
Case 3: if f(n) = (nlog
b
a +
) for some > 0, and if
af(n/b) ≤ cf(n) for some c < 1 and all sufficiently large n, then:
T(n) = (f(n))
)
(
)
( n
f
b
n
aT
n
T
regularity condition
27. 28
Examples
T(n) = 2T(n/2) + n
a = 2, b = 2, log22 = 1
Compare nlog
2
2
with f(n) = n
f(n) = (n) Case 2
T(n) = (nlgn)
28. 29
Examples
T(n) = 2T(n/2) + n2
a = 2, b = 2, log22 = 1
Compare n with f(n) = n2
f(n) = (n1+
) Case 3 verify regularity cond.
a f(n/b) ≤ c f(n)
2 n2
/4 ≤ c n2
c = ½ is a solution (c<1)
T(n) = (n2
)
29. 30
Examples (cont.)
T(n) = 2T(n/2) +
a = 2, b = 2, log22 = 1
Compare n with f(n) = n1/2
f(n) = O(n1-
) Case 1
T(n) = (n)
n
30. 31
Examples
T(n) = 3T(n/4) + nlgn
a = 3, b = 4, log43 = 0.793
Compare n0.793
with f(n) = nlgn
f(n) = (nlog
4
3+
) Case 3
Check regularity condition:
3(n/4)lg(n/4) ≤ (3/4)nlgn = c f(n), c=3/4
T(n) = (nlgn)
31. 32
Examples
T(n) = 2T(n/2) + nlgn
a = 2, b = 2, log22 = 1
• Compare n with f(n) = nlgn
– seems like case 3 should apply
• f(n) must be polynomially larger by a factor of n
• In this case it is only larger by a factor of lgn
![4
Recurrent Algorithms
BINARY-SEARCH
• for an ordered array A, finds if x is in the array A[lo…hi]
Alg.: BINARY-SEARCH (A, lo, hi, x)
if (lo > hi)
return FALSE
mid (lo+hi)/2
if x = A[mid]
return TRUE
if ( x < A[mid] )
BINARY-SEARCH (A, lo, mid-1, x)
if ( x > A[mid] )
BINARY-SEARCH (A, mid+1, hi, x)
12
11
10
9
7
5
3
2
1 2 3 4 5 6 7 8
mid
lo hi](https://image.slidesharecdn.com/recurrences-250812080652-f409ee5a/85/Recurrences-DATA-ANALYSIS-ALGORITHMS-ppt-4-320.jpg)
![5
Example
• A[8] = {1, 2, 3, 4, 5, 7, 9, 11}
– lo = 1 hi = 8 x = 7
mid = 4, lo = 5, hi = 8
mid = 6, A[mid] = x
Found!
11
9
7
5
4
3
2
1
11
9
7
5
4
3
2
1
1 2 3 4 5 6 7 8
8
7
6
5](https://image.slidesharecdn.com/recurrences-250812080652-f409ee5a/85/Recurrences-DATA-ANALYSIS-ALGORITHMS-ppt-5-320.jpg)
![6
Another Example
• A[8] = {1, 2, 3, 4, 5, 7, 9, 11}
– lo = 1 hi = 8 x = 6
mid = 4, lo = 5, hi = 8
mid = 6, A[6] = 7, lo = 5, hi = 5
11
9
7
5
4
3
2
1
11
9
7
5
4
3
2
1
1 2 3 4 5 6 7 8
11
9
7
5
4
3
2
1 mid = 5, A[5] = 5, lo = 6, hi = 5
NOT FOUND!
11
9
7
5
4
3
2
1
low high
low
low
high
high](https://image.slidesharecdn.com/recurrences-250812080652-f409ee5a/85/Recurrences-DATA-ANALYSIS-ALGORITHMS-ppt-6-320.jpg)
![7
Analysis of BINARY-SEARCH
Alg.: BINARY-SEARCH (A, lo, hi, x)
if (lo > hi)
return FALSE
mid (lo+hi)/2
if x = A[mid]
return TRUE
if ( x < A[mid] )
BINARY-SEARCH (A, lo, mid-1, x)
if ( x > A[mid] )
BINARY-SEARCH (A, mid+1, hi, x)
• T(n) = c +
– T(n) – running time for an array of size n
constant time: c2
same problem of size n/2
same problem of size n/2
constant time: c1
constant time: c3
T(n/2)](https://image.slidesharecdn.com/recurrences-250812080652-f409ee5a/85/Recurrences-DATA-ANALYSIS-ALGORITHMS-ppt-7-320.jpg)
