Internet peering, with annotations
4 likes2,464 views
The document discusses the evolution of internet infrastructure, focusing on the competitive and cooperative dynamics among ISPs that form the backbone of the internet. It outlines how the initial single backbone network transitioned to a diverse ecosystem of thousands of major ISPs, peering agreements, and the introduction of content delivery networks. The document highlights the historical context and key developments that have shaped the current state of internet transit and peering, revealing that no entity has managed to monopolize the internet backbone despite attempts to establish cartels.
Internet peering, with annotations
- 1. Why care? Peering and Transit are li3le understood and yet this is the heart of Internet infrastructure economics and it is the growth of Internet infrastructure that ul:mately funds most of us here at OFC. I going to describe how thousands of independent organiza:ons compete but also exchange traffic and con:nually grow an ever more distributed Internet infrastructure. 1
- 2. There are more than 3 billion Internet users and, with the advent of very low cost Android smart phones ($24 in India in Feb 2014), it’s likely that number will double in just a few years. There are also tens of millions of local networks (perhaps more than 100 million based just on the number of WiFi routers that have been sold). They connect to the Internet through tens of thousands of ISPs which may be classified as Access, Aggrega:on or Backbone, or as Local, Regional, Na:onal or Interna:onal but, of course, many ISPs cross these boundaries. For today, I’m going to focus on the 6000 or so major ISPs that form today’s Internet backbone. But to understand today’s complex environment, it’s useful to see how it emerged. 2
- 3. 25 years ago, there was only one backbone. It was run by the Na:onal Science Founda:on for the benefit of various researchers and government agencies. Regional networks connected to the NSF backbone but, with only one backbone, there was only one source of addressing and of ul:mate rou:ng decisions. 3
- 4. As other, commercial networks grew up, they interconnected with the NSFNET to exchange email and data files. They also found other ways to exchange data among themselves, but s:ll relied on the NSFnet as the ul:mate authority on addressing and rou:ng. 4
- 5. With the advent of the World Wide Web and the Mosaic browser, Internet growth accelerated and the NSF sought a way to get out of the backbone business. Part of this required development of a new rou:ng protocol (BGP, on which more later) which went on within the IETF between 1991-‐1994. Part of this required establishing four Network Access Points (NAPs) where backbone providers would exchange traffic des:ned for other backbones. Of course, each backbone provider had its own network that enabled all connected users and content providers to communicate with one another. However, users were not interested in communica:ng just with just those other users connected to the same backbone provider. They wanted to communicate with any user and any content provider, regardless of backbone provider. To offer universal connec:vity, backbone providers interconnected at NAPs (and elsewhere) to exchange traffic des:ned for each other’s users. It is these interconnec:ons that make the Internet the “network of networks” that it is today. Finally, in April 1995, the NSF stopped providing backbone services and the commercial Internet was born. 5
- 6. In order to provide complete Internet access, the backbone providers had to exchange traffic with each other. What’s more, the NSFNET backbone had facilitated open traffic exchange at many levels, so there were many peering agreements at first. But the Internet was also growing rapidly, requiring significant capital investments. At a minimum, investors wanted to see a path to a return on their investment. 6
- 7. And, with just six-‐seven full backbones networks in existence, the backbone ISPs began to realize they had the makings of a cartel. 7
- 8. As a cartel, none of the backbone operators had to provide free peering to regional, local or other smaller networks. Instead they could sell them “Internet Transit” service – a service that delivers packets to the rest of the Internet. Gradually (and some:mes abruptly), peering rules became quite exclusive. To peer with the backbone, you had to be present at all major NAPs, you had to have a significant amount of traffic and that traffic had to be roughly symmetric. This had an immediate impact on many Tier 2 operators, some of which were growing more rapidly than the backbones. 8
- 9. De-‐peering also impacted cable companies, several major content hos:ng networks and some large savvy content providers. These folks realized that, even if they had to buy “Internet Transit” from a backbone provider, they could reduce what they paid the backbone providers by exchanging traffic among themselves. 9
- 10. By 2002, donut peering had emerged. The Tier 2 ISPs, cable companies and content providers had built a ring around the cartel, largely rendering the original cartel irrelevant. 10
- 11. Indeed, many Tier 2 providers now had interna:onal networks and offered lower latency &/or be3er pricing. By the early 2000s, the Internet was substan:ally more distributed. 11
- 12. The third wave, which started in the early 2000s and is s:ll evolving today, was the advent of Content Distribu:on Networks (CDNs). CDNs may have limited, private or no communica:ons infrastructure of their own, instead they distribute content servers in what is effec:vely an overlay network. Akamai and Limelight created early CDNs. Today, Google, Amazon and Level 3 also run content distribu:on networks and Nejlix has begun deploying their own CDN. 12
- 13. Typically, major CDNs supply their servers and remotely manage them, but local ISPs install them and pay for electricity and rack space. This is good business for the local ISP as it reduces latency for their customers and reduces the amount of upstream Internet transit service they must pay for. 13
- 14. The past 20 years have seen enormous turbulence among those providing the core of the Internet. The original backbone networks have survived, but their ownership has gone through a series of bankruptcies, mergers and acquisi:ons. Meanwhile the number of networks par:cipa:ng in the Internet backbone has grown from 6 to over 6000. 14
- 15. I’ve been bandying around the terms “Peering” and “Internet Transit.” Let me explain exactly how they differ. Internet Transit is a service where the upstream ISP commits to deliver traffic to any valid Internet address. It’s typically priced in $/Mbps/Month and the Mbps of traffic is determined by measuring traffic levels every five minutes and then compu:ng the 95th percen:le of all those measurements during the month. Now suppose I’m ISP1. I have a router in a regional data center where I buy Internet Transit services, but I no:ce that 4% of my traffic is to my compe:tor, ISP2, and he happens to have a router in the same regional data center just a few hundred feet away from mine. He’s my compe:tor, but we could each save 4% of our monthly bills for Internet transit if we agree to locally exchange the traffic that’s des:ned for each other’s networks. 15
- 16. No:ce that we’re only exchanging traffic that originates with a customer of one ISP and terminates with a customer of the other peered ISP. 16
- 17. ISP2 may have other connec:ons to other ISPs, but these are not involved (or even visible) to the peering arrangement with ISP1. That’s the key difference. Peering is traffic exchange involving only those addresses that are served by the two peers. Transit involves handling packets that will be passed off to one or more addi:onal networks. 17
- 18. But whether it’s peering or transit, what is actually exchanged and how does it work? Here things are remarkably stable. Operators may exchange other kinds of traffic (MPLS, Carrier Ethernet) for other services, but for Internet traffic, they exchange IP packets (mostly IPv4) and they nego:ate routes using Border Gateway Protocol (BGP). IPv4 is essen:ally unchanged for over 30 years and the current version of BGP has had only minor tweaks since it was deployed 20 years ago. Business arrangements have been turbulent, but the technology has been remarkably stable. 18
- 19. To get a be3er understanding of BGP, suppose I’m running BGP on my edge router there on the lel. There are two ISPs I wish to exchange traffic with (either peering or transit). In par:cular, I’m interested in gemng traffic to address blocks A, B & C. My router starts by establishing BGP sessions with the edge routers at each ISP. 19
- 20. Once the sessions are up, I get an announcement from the edge router at ISP1 saying it’s prepared to deliver traffic to address block A over a route that has three hops and traffic for address block B over a route that has one hop. 20
- 21. This is followed by an announcement from ISP2 saying they can deliver traffic to address block B in two hops or to address block C in two hops. Now, I have to make some decisions. 21
- 22. First these announcements come from other organiza:ons who may or may not be competent. Should I believe ISP1 when he says he can deliver traffic to address block B in just one hop? A classic example of mistakes that can happen occurred in Feb 2008 when the government of Pakistan told Pakistan Telecom to block traffic to YouTube because YouTube was hos:ng blasphemous videos. The engineers at Pakistan Telecom complied by crea:ng a very specific route for just the YouTube addresses (part of a larger Google address block). Request packets that matched this specific route were sent to a “black hole server,” i.e. a server that dropped each packet it received. Unfortunately, this black hole route leaked out to the large interna:onal carrier, Hong Kong-‐based PCCW. PCCW didn’t have route filtering in place on this par:cular link and they passed the black hole route around the world. Over 90 major ISPs erroneously accepted this route and for more than two hours YouTube was dark while almost all the world’s YouTube requests went to the black hole server in Pakistan. So you can’t always trust your neighbor, however competent they may have seemed in the past. There are many addi:onal considera:ons. For example, certain routes may have preferen:al pricing up to a certain commitment level but become expensive at higher traffic levels. So the choice of which adver:sed route to use can involve some quite complex considera:ons. 22
- 23. To give you a sense of the business trade offs that go on, I have two examples. The first is a friend of mine who formed a fixed wireless ISP in southeastern Illinois a few years ago. Because he was located in farm country, the only way he could get an Internet connec:on was by buying Internet Transit service (called Direct Internet Access or DIA) from Ameritech (now AT&T) the local telephone monopoly. His price was more than 100x what Internet Transit would have cost him in Chicago, but there were no compe:ng fiber routes through his area and even if he’d been close to a long distance fiber route (say between Chicago and St Louis), local connec:ons to long distance fiber are extremely expensive or, more olen, just not available. Once his business was up and running, my friend spent many days driving to and from Chicago looking for tall buildings and talking to building owners. Eventually he build a series of four wireless links (totaling more than 70 miles) which connected him to Chicago. In Chicago, he signed up for a monthly recurring charge for rack space, for roof rights on the Chicago data center and for a cable from his rack to their “meet me room.” He’d also promised free high speed Internet service to three building owners, downstate, who gave him roof access on the route to Chicago. But now that he was connected in Chicago, he could purchase Internet transit from any of a dozen compe:ng carriers (at a :ny frac:on of what he was paying AT&T). Although he had spent nearly $100K (and untold man hours) pumng this wireless route together, he figured his payback was 9 weeks. Loca:on ma3ers! The second thing that happened was, as his total traffic grew he began to qualify for peering with major content providers like Google and Akamai. This cut further cut his costs for Internet transit. 23
- 24. The second example is only approximate, but representa:ve. I don’t have the actual numbers on YouTube’s traffic or their costs during the 20 months between their founding in Feb 2005 and their purchase by Google in Oct-‐Nov 2006, but I can tell you that one of their early employees was a “peering coordinator” who showed up at NANOG mee:ngs early in 2006. In early 2006, there was already a great interest in peering with YouTube. By the summer of 2006, YouTube was the 5th most traffic’d website in the world. They were s:ll only peering in Palo Alto, but anyone with a router in Palo Alto was interested in offloading their YouTube traffic. And any Tier one carrier that didn’t peer with YouTube would quickly find traffic ra:os going unbalanced on links where they handed off YouTube traffic to someone who was peering with YouTube. I’m not showing YouTube’s costs going to zero, but they clearly did not increase (and likely went down) as YouTube’s traffic grew! 24
- 25. Bill Woodcock and Vijay Adhikari of Packet Clearing House did a very comprehensive survey of backbone ISPs in 2011 gemng a remarkable 86% response rate. All the internal indica:ons are this survey yielded very high quality data. Several interes:ng things emerged from this data. Most notably, many operators publish a set of peering requirements, and these typically include an NDA. But if you meet the requirements, there are no formal contracts! These are handshake agreements. 25
- 26. One interes:ng thing was, to the extent there are contracts between operators in different countries, for example the NDAs, the choice of governing law always favors the country with stable ins:tu:ons, minimum corrup:on and a func:oning judiciary. 26
- 27. In terms of how the Internet backbone is evolving, the most interes:ng thing to emerge was the rise of mul:-‐lateral peering. These are arrangements that started in Asia and selected loca:ons in Europe. We haven’t seen this in the US yet, but there is an organiza:on, “open-‐ix.org,” backed by Google and Amazon among others, that is trying to foster the spread of mul:-‐lateral peering. 27
- 28. Mul:-‐lateral peering drama:cally reduces the number of BGP sessions one must configure and manage, thus facilita:ng more peering. With bi-‐lateral peering, there is a separate BGP session for every peer. 28
- 29. In mul:-‐lateral peering, one organiza:on – perhaps a co-‐op or a vendor – provides a single route server. Each par:cipant establishes a single BGP session to this server. Typically, the route server includes session-‐specific configura:on which allows you some of the flexibility you would have had with N bi-‐lateral peering sessions but, to get started, you can ignore all that and just establish one simple BGP session that reaches hundreds of peers. 29
- 30. This graph shows the number of IP addresses handled by various carriers as a func:on of how many peering agreements those carriers have. You can see that some of the original Tier 1s are s:ll visible in the upper lel, but otherwise, the Internet backbone is very distributed. And this graph is based on addresses handled, not on traffic carried. 30
- 31. When we look at traffic, the top ISPs are quite distributed. Also, we can see what happens when the large carrier (Level 3 at the top) buys the second largest carrier (Global Crossing in grey just below) as happened in April 2012. Both networks immediately saw a drop in traffic as customers who wanted redundant connec:ons dropped one of their connec:ons to the now merged business. Then over :me, both networks see further drops in traffic as the rest of the players rearrange their networks. Also, note that this traffic diagram only deals with ISPs that offer Internet transit services. The second largest network in the world, by traffic, is Google. If Google were shown on this graph, it would appear between the black and grey lines. So the Internet is very distributed and, as Rensys notes in their report, the rela:ve market share of the backbone carriers as a group has been falling over the past decade. In ~20 years of the commercial Internet, no one has been able to gain control of the Internet backbone. In the 1990s, the original gang of ~six backbone providers thought they had an oligopoly (a cartel), but by 2002, second :er backbones used "donut peering” to eliminate the original :er ones’ leverage. Since 2000, we've seen the emergence of mul:ple CDNs (Akamai, Level 3, Google, Limelight, plus Amazon, Nejlix, and others in the making) which have further diluted any a3empt to monopolize the backbone. Also over the past 20+ years, we've seen an explosion in the number of buildings where some kind of peering takes place. In short, no one has been able to monopolize the Internet backbone. Now we’re seeing the emergence of mul:-‐lateral peering and even more backbone par:cipants. 31
- 32. The Internet backbone is a very interes:ng phenomenon. It’s essen:ally unregulated. IANA (the body that supervises the assignment of addresses and other protocol number assignments) provides only coordina:on. If IANA withheld or manipulated assignments, their func:on could be quickly and informally bypassed. Recently we’ve heard a lot about “regula:ng the Internet” especially since the revela:ons of NSA spying. But most such discussion is happening without any understanding of how the Internet backbone actually works. Forecasts are iffy, but the current system is extremely successful and extremely robust, so I am op:mis:c the Internet will con:nue to grow, indefinitely. 32
- 33. 33