Asymptotic Growth of Functions
1 like2,254 views
This document introduces asymptotic notation used to analyze the runtime of algorithms. Big O notation describes upper bounds on function growth, while Ω notation describes lower bounds. Functions are asymptotically equivalent (Θ) if they have matching upper and lower bounds. Limits can be used to establish relationships between asymptotic classes in some cases, but not always - examples show membership in a class does not necessarily imply limits exist.
Asymptotic Growth of Functions
- 1. 1 CMPS 102 Introduction to Analysis of Algorithms Fall 2003 Asymptotic Growth of Functions We introduce several types of asymptotic notation which are used to compare the performance and efficiency of algorithms. As we’ll see, the asymptotic run time of an algorithm gives a simple, and machine independent, characterization of it’s complexity. Definition Let )(ng be a function. The set ))(( ngO is defined as })()(0:,0,0|)({))(( 00 ncgnfnnncnfngO ≤≤≥∀>∃>∃= . In other words, ))(()( ngOnf ∈ if and only if there exist positive constants c, and 0n , such that for all 0nn ≥ , the inequality )()(0 ncgnf ≤≤ is satisfied. We say that )(nf is Big O of )(ng , or that )(ng is an asymptotic upper bound for )(nf . We often abuse notation slightly by writing ))(()( ngOnf = to mean ))(()( ngOnf ∈ . Actually ))(()( ngOnf ∈ is also an abuse of notation. We should really write )(gOf ∈ since what we have defined is a set of functions, not a set of numbers. The notational convention ))(( ngO is useful since it allows us to refer to the set )( 3 nO say, without having to introduce a function symbol for the polynomial 3 n . Observe that if ))(()( ngOnf = then )(nf is asymptotically non-negative, i.e. )(nf is non-negative for all sufficiently large n, and likewise for )(ng . We make the blanket assumption from now on that all functions under discussion are asymptotically non-negative. In practice we will be concerned with integer valued functions of a (positive) integer n ( ++ → ZZ:g ). However, in what follows, it is useful to consider n to be a continuous real variable taking positive values and g to be real valued function ( ++ → RR:g ). Geometrically ))(()( ngOnf = says: )(ncg )(nf 0n
- 2. 2 Example )30010(10040 2 ++=+ nnOn . Observe that 30010100400 2 ++≤+≤ nnn for all 20≥n , as can be easily verified. Thus we may take 200 =n and 1=c in the definition. 300102 ++ nn 300 10040 +n 100 10 20 Note that in this example, any value of 0n greater than 20 will also work, and likewise any value of c greater than 1 works. In general if there exist positive constants 0n and c such that )()(0 ncgnf ≤≤ for all 0nn ≥ , then infinitely many such constants also exist. In order to prove that ))(()( ngOnf = it is not necessary to find the smallest possible 0n and c making the )()(0 ncgnf ≤≤ true. It is only necessary to show that at least one pair of such constants exist. Generalizing the last example, we will show that )( 2 edncnOban ++=+ for any constants a-e, and in fact ))(()( nqOnp = whenever )(np and )(nq are polynomials with )deg()deg( qp ≤ . Definition Let )(ng be a function and define the set ))(( ngΩ to be })()(0:,0,0|)({))(( 00 nfncgnnncnfng ≤≤≥∀>∃>∃=Ω . We say )(nf is big Omega of )(ng , and that )(ng is an asymptotic lower bound for )(nf . As before we write ))(()( ngnf Ω= to mean ))(()( ngnf Ω∈ . The geometric interpretation is: )(nf )(ncg 0n
- 3. 3 Lemma ))(()( ngOnf = if and only if ))(()( nfng Ω= . Proof: If ))(()( ngOnf = then there exist positive numbers 1c , 1n such that )()(0 1 ngcnf ≤≤ for all 1nn ≥ . Let 12 /1 cc = and 12 nn = . Then )()(0 2 ngnfc ≤≤ for all 2nn ≥ , proving ))(()( nfng Ω= . The converse is similar and we leave it to the reader. /// Definition Let )(ng be a function and define the set ))(())(())(( ngngOng Ω∩=Θ . Equivalently })()()(0:,0,0,0|)({))(( 210021 ngcnfngcnnnccnfng ≤≤≤≥∀>∃>∃>∃=Θ . We write ))(()( ngnf Θ= and say the )(ng is an asymptotically tight bound for )(nf , or that )(nf and )(ng are asymptotically equivalent. We interpret this geometrically as: )(2 ngc )(nf )(1 ngc 0n Exercise Prove that if c is a positive constant, then ))(()( nfncf Θ= . Exercise Prove that ))(()( ngnf Θ= if and only if ))(()( nfng Θ= . Example Prove that ( )nn Θ=+10 . Proof: According to the definition, we must find positive numbers 021 ,, ncc , such that the inequality ncnnc 21 100 ≤+≤≤ holds for all 0nn ≥ . Pick 11 =c , 22 =c , and 100 =n . Then if 0nn ≥ we have: 010 ≤− and n≤10 ∴ n)11(10 −≤− and n)12(10 −≤ ∴ nc )1(10 2 1−≤− and nc )1(10 2 2 −≤ ∴ 102 1 +≤ nnc and ncn 2 210 ≤+ , ∴ ncnnc 2 2 2 1 10 ≤+≤ , ∴ ncnnc 21 10 ≤+≤ , as required. ///
- 4. 4 The reader may find our choice of values for the constants 021 ,, ncc in this example somewhat mysterious. Adequate values for these constants can usually be obtained by working backwards algebraically from the inequality to be proved. Notice that in this example there are many valid choices. For instance one checks easily that 2/11 =c , 2/32 =c , and 200 =n work equally well. Exercise Let a, b be real numbers with 0>b . Prove directly from the definition (as above) that )()( bb nan Θ=+ . (In what follows we learn a much easier way to prove this.) Lemma If )()( nhnf ≤ for all sufficiently large n, and if ))(()( ngOnh = , then ))(()( ngOnf = . Proof: The above hypotheses say that there exist positive numbers c and 1n such that )()( ncgnh ≤ for all 1nn ≥ . Also there exists 2n such that )()(0 nhnf ≤≤ for all 2nn ≥ . (Recall )(nf is assumed to be asymptotically non-negative.) Define ),max( 210 nnn = , so that if 0nn ≥ we have both 1nn ≥ and 2nn ≥ . Thus 0nn ≥ implies )()(0 ncgnf ≤≤ , and therefore ))(()( ngOnf = . /// Exercise Prove that if )()()( 21 nhnfnh ≤≤ for all sufficiently large n, where ))(()(1 ngnh Ω= and ))(()(2 ngOnh = , then ))(()( ngnf Θ= . Example Let 1≥k be a fixed integer. Prove that )( 1 1 + = Θ= k n i k ni . Proof: Observe that )( 11 11 ++ == ==⋅=≤ kkk n i k n i k nOnnnni , and )()2/1()2/)(2/()2/(2/)2/( 111 2/2/1 +++ === Ω==≥⋅≥≥≥ kkkkk n ni k n ni k n i k nnnnnnnii . By the result of the preceding exercise, we conclude )( 1 1 + = Θ= k n i k ni . /// When asymptotic notation appears in a formula such as )()2/(2)( nnTnT Θ+= we interpret )(nΘ to stand for some anonymous function in the class )(nΘ . For example )(31243 2323 nnnnn Θ+=+−+ . Here )( 2 nΘ stands for 124 2 +− nn , which belongs to the class )( 2 nΘ . The expression = Θ n i i1 )( can be puzzling. On the surface it stands for )()3()2()1( nΘ++Θ+Θ+Θ , which is meaningless since )constant(Θ consists of all functions which are bounded above by some constant. We interpret )(iΘ in this expression to stand for a single function )(if in the class )(iΘ , evaluated at ni ,,3,2,1= . Exercise Prove that )()( 2 1 ni n i Θ=Θ= . The left hand side stands for a single function )(if summed for ni ,,3,2,1= . By the previous exercise it is sufficient to show that )()()( 211 nhifnh n i ≤≤ = for all sufficiently large n, where )()( 2 1 nnh Ω= and )()( 2 2 nOnh = .
- 5. 5 Definition })()(0:,0,0|)({))(( 00 ncgnfnnncnfngo <≤≥∀>∃>∀= . We say that )(ng is a strict Asymptotic upper bound for )(nf and write ))(()( ngonf = as before. Lemma ))(()( ngonf = if and only if 0 )( )( lim = ∞→ ng nf n . Proof: Observe that ))(()( ngonf = if and only if c ng nf nnnc <≤≥∀>∃>∀ )( )( 0:,0,0 00 , which is the very definition of the limit statement 0 )( )( lim = ∞→ ng nf n . /// Example )()lg( non = since 0 )lg( lim = ∞→ n n n . (Apply l’Hopitals rule.) Example )( nk bon = for any 0>k and 1>b since 0lim = ∞→ n k n b n . (Apply l’Hopitals rule k times.) In other words, any exponential grows strictly faster than any polynomial. By comparing definitions of ))(( ngo and ))(( ngO one sees immediately that ))(())(( ngOngo ⊆ . Also no function can belong to both ))(( ngo and ))(( ngΩ , as is easily verified (exercise). Thus ∅=Ω∩ ))(())(( ngngo , and therefore ))(())(())(( ngngOngo Θ−⊆ . Definition })()(0:,0,0|)({))(( 00 nfncgnnncnfng <≤≥∀>∃>∀=ω . Here we say that )(ng is a strict asymptotic lower bound for )(nf and write ))(()( ngnf ω= . Exercise Prove that ))(()( ngnf ω= if and only if ∞= ∞→ )( )( lim ng nf n . Exercise Prove ∅=∩ ))(())(( ngOngω , whence ))(())(())(( ngngng Θ−Ω⊆ω . The following picture emerges: ))(( ngO ))(( ngΩ ))(( ngo ))(( ngΘ ))(( ngω
- 6. 6 Lemma If L ng nf n = ∞→ )( )( lim , where ∞<≤ L0 , then ))(()( ngOnf = . Proof: The definition of the above limit is εε <−≥∀>∃>∀ L ng nf nnn )( )( :,0,0 00 . Thus if we let 1=ε , there exists a positive number 0n such that for all 0nn ≥ : 1 )( )( <− L ng nf ∴ 1 )( )( 1 <−<− L ng nf ∴ 1 )( )( +< L ng nf ∴ )()1()( ngLnf ⋅+< . Now take 1+= Lc in the definition of O , so that ))(()( ngOnf = as claimed. /// Lemma If L ng nf n = ∞→ )( )( lim , where ∞≤< L0 , then ))(()( ngnf Ω= . Proof: The limit statement implies L nf ng n ′= ∞→ )( )( lim , where LL /1=′ and hence ∞<′≤ L0 . By the previous lemma ))(()( nfOng = , and therefore ))(()( ngnf Ω= . /// Exercise Prove that if L ng nf n = ∞→ )( )( lim , where ∞<< L0 , then ))(()( ngnf Θ= . Although ))(( ngo , ))(( ngω , and a certain subset of ))(( ngΘ are characterized by limits, the full sets ))(( ngO , ))(( ngΩ , and ))(( ngΘ have no such characterization as the following examples show. Example A Let nng =)( and nnnf ⋅+= ))sin(1()( . )(2 ng )(nf Clearly ))(()( ngOnf = , but )sin(1 )( )( n ng nf += , whose limit does not exist. This example shows that the containment ))(())(())(( ngngOngo Θ−⊆ is in general strict since ))(()( ngnf Ω≠ (exercise). Therefore ))(()( ngnf Θ≠ , so that ))(())(()( ngngOnf Θ−∈ . But ))(()( ngonf ≠ since the limit does not exist.
- 7. 7 Example B Let nng =)( and nnnf ⋅+= ))sin(2()( . )(3 ng )(nf )(ng Since nnnn 3))sin(2( ≤⋅+≤ for all 0≥n , we have ))(()( ngnf Θ= , but )sin(2 )( )( n ng nf += whose limit does not exist. Exercise Find functions )(nf and )(ng such that ))(())(()( ngngnf Θ−Ω∈ , but )( )( lim ng nf n ∞→ does not exist (even in the sense of being infinite), so that ))(()( ngnf ω≠ . The preceding limit theorems and counter-examples can be summarized in the following diagram. Here L denotes the limit )( )( lim ng nf L n ∞→ = , if it exists. ))(( ngO ))(( ngΩ ))(( ngΘ ))(( ngo ))(( ngω 0=L ∞<< L0 ∞=L Ex A Ex B In spite of the above counter-examples, the preceding limit theorems are a very useful tool for establishing asymptotic comparisons between functions. For instance recall the earlier exercise to show )()( bb nan Θ=+ for real numbers a, and b with 0>b . The result follows immediately from 111lim )( lim ==+= + ∞→∞→ b b nb b n n a n an , since ∞<<10 .
- 8. 8 Exercise Use limits to prove the following: a. )()lg( 2 nonn = (here )lg(n denotes the base 2 logarithm of n.) b. )(2 105 nn n ω= . c. If )(nP is a polynomial of degree 0≥k , then )()( k nnP Θ= . d. ))(())(()( nfnfonf Θ=+ . (One can always disregard lower order terms) e. )()(log ε non k = for any 0>k and 0>ε . (Polynomials grow faster than logs.) f. )( n bon =ε for any 0>ε and 1>b . (Exponentials grow faster than polynomials.) There is an analogy between the asymptotic comparison of functions )(nf and )(ng , and the comparison of real numbers x and y. ))(()( ngOnf = ~ yx ≤ ))(()( ngnf Θ= ~ yx = ))(()( ngnf Ω= ~ yx ≥ ))(()( ngonf = ~ yx < ))(()( ngnf ω= ~ yx > Note however that this analogy is not exact since there exist pairs of functions which are not comparable, while any two real numbers are comparable. (See problem 3-2c, p.58.)