Time complexity
- 1. Time Complexity UC Berkeley Fall 2004, E77 http://jagger.me.berkeley.edu/~pack/e77 Copyright 2005, Andy Packard. This work is licensed under the Creative Commons Attribution-ShareAlike License. To view a copy of this license, visit http://creativecommons.org/licenses/by-sa/2.0/ or send a letter to Creative Commons, 559 Nathan Abbott Way, Stanford, California 94305, USA.
- 2. Time complexity of algorithms Dependency of – the time it takes to solve a problem – as a function of the problem dimension/size Examples: – Sorting a list of length n – Searching a list of length n – Multiplying a n×n matrix by an n×1 vector Time to solve problem might depend on data – Average-case time – Best-case time • data is well suited for algorithm (can’t be counted on) – Worst-case time • data is such that algorithm performs poorly (time-wise) Worst-Case gives an upper bound as to how much time will be needed to solve any instance of the problem.
- 3. Example: N-by-N matrix, N-by-1 vector, multiply Total = c1N+c2N+N(c3+c2N+N(c4+c5)+c5) = (c2+c4+c5)N2 + (c1+c2+c3+c5)N = c6N2 + c7N N times Y = zeros(N,1); initialize space, c1N initialize “for” loop, c2N for i=1:N Scalar assignment, c3 Y(i) = 0.0; initialize “for” loop, c2N for j=1:N c4 Y(i) = Y(i) + A(i,j)*x(j); End of loop, return/exit, c5 end End of loop, return/exit, c5 end N times (3 accesses, 1 add, 1 multiply)
- 4. Asymptotic Time complexity of algorithms Dependency of – the time it takes to solve a problem – as a function of the problem dimension/size but – formula may only be valid for “large” problems So, we keep track of “growth rates” of the computational workload as a function of problem dimension
- 5. Order, called “big O” notation Given two functions, f and g, say that “f is of order g” if – there is a constant c, and – a value x0 such that f ( x ) ≤ c ⋅ g ( x ) for all x > x0 So, apart from a fixed multiplicative constant, the function g is an – upper bound on the function f – valid for large values of its argument. Notation: write f = O( g ) to mean “f is of order g”. Sometimes write f ( x ) arguments are labled. = O( g ( x ) ) to remind us what the
- 6. Not equality,but “belongs to a class” Recall that f ( n ) = O( g ( n ) ) means that – there is a constant c, and – a value n0 such that f ( n ) ≤ c ⋅ g ( n ) for all n > n0 The = sign does not mean equality! It means that f is an element of the set of functions which are eventually bounded by (different) constant multiples of g. Therefore, it is correct/ok to write 3n = O( n ) , ( ) 3n = O n 2 , ( 3n = O n 3 ⋅ log n )
- 7. Big O: Examples Example: Note: f ( n ) = 31 g ( n) = 1 for all n > 0, 31 ≤ 31 f ( n ) 31⋅ g (n) For all n, f is bounded above by 31g Write or f ( n ) = O( g ( n ) ) 31 = O(1)
- 8. Big O: Examples Example: f ( n ) = 4n 2 + 7 n + 12 g ( n) = n 2 for n > 8, 4n 2 + 7 n + 12 < 5n 2 Note: f ( n) For large enough n f is bounded above by 5g Write f ( n ) = O( g ( n ) ) f = O( g ) or 4n 2 + 7n + 12 = O n 2 or ( ) 5 g ( n) 1200 1000 800 600 400 200 0 0 5 10 15
- 9. Big O: Relationships Suppose f1 and f2 satisfy: There is a value n0 f1 ( n ) ≤ f 2 ( n ) for all n > n0 Then f1 ( n ) + f 2 ( n ) ≤ 2 f 2 ( n ) for all n > n0 Hence f1 + f 2 = O( f 2 ) Example 3: 27 log( n ) + 19 + n = O( n ) f1 ( n ) f 2 ( n) Generalization: f1 = O( f 2 ) ⇒ f1 + f 2 = O( f 2 )
- 10. Big O: Relationships Suppose positive functions f1, f2, g1, g2, satisfy f1 = O( g1 ) and Then f1 ⋅ f 2 = O( g1 ⋅ g 2 ) f 2 = O( g 2 ) Why? There exist c1, c2, n1, n2 such that f i ( n ) ≤ ci ⋅ g i ( n ) for all n > ni Take c0=c1c2, n0=max(n1,n2). Multiply to give f1 ( n ) f ( n ) 2 ≤ c0 ⋅ g1 ( n ) g 2 ( n ) for all n > n0 Example: (12n 2 ) ( ) + 2 log n ( 27 log( n ) + 19 + n ) = O n 3 f 2 ( n) f1 ( n )
- 11. Asymptotic: Relationships Obviously, for any positive function g g = O( g ) Let β be a positive constant. Then β ⋅ g = O( g ) as well. ( ) Example 4: 13n = O n 3 3 Message: Bounding of growth rate. If n doubles, then the bound grows by 8. Example 5: log n = O( log n ) 512
- 12. N times Y = zeros(N,1); initialize space, c1N initialize “for” loop, c2N for i=1:N Scalar assignment, c3 Y(i) = 0.0; initialize “for” loop, c2N for j=1:N c4 Y(i) = Y(i) + A(i,j)*x(j); End of loop, return/exit, c5 end End of loop, return/exit, c5 end N times Example: N-by-N matrix, N-by-1 vector, multiply Total = c6N2 + c7N •Problem size affects (is, in fact) N •Processor speed, processor and language architecture, ie., technology, affect c6 and c7 Hence, “this algorithm of matrix-vector multiply has O(N2) complexity.”
- 13. Time complexity familiar tasks Task Matrix/vector multiply Getting a specific element from a list Dividing a list in half, dividing one halve in half, etc Binary Search Scanning (brute force search) a list Nested for loops (k levels) MergeSort BubbleSort Generate all subsets of a set of data Generate all permutations of a set of data Growth rate O(N2) O(1) O(log2N) O(log2N) O(N) O(Nk) O(N log2N) O(N2) O(2N) O(N!)