21092018-C4E-What's Next for the Net? Security, Reliability, Capability, Performance, Flexibility and More[1]]
Download as PPTX, PDF1 like197 views
The document provides an introduction to routing and interconnection on the internet. It discusses how the internet is made up of interconnected networks and how routing works through peering relationships and transit agreements between networks. It also introduces internet exchange points (IXPs) as locations where networks can peer with each other to exchange traffic. Visualizations of network diagrams are also presented to help illustrate how individual networks interconnect both domestically and globally on the internet infrastructure.
21092018-C4E-What's Next for the Net? Security, Reliability, Capability, Performance, Flexibility and More[1]]
- 1. Routing/Interconnection: An Introduction “The Internet works by interconnecting networks” “This is the IETF RFCs in action” University of Cyprus Sepember 2018 Martin J. Levy, Network Strategy Cloudflare
- 2. Introduction & Agenda The word “Internet” Interconnected networks The relationship between individual networks IP transit IP peering Three key topics An introduction to peering The creation of IXPs (Internet Peering Points) Visualizing interconnected networks
- 3. The word “Internet” - How routing works Technically – a very complex subject The Internet is a collection of networks No network stands alone Interconnections are required Robustness can be created Multi-homing (more than one transit) Peering between “like” networks Diversity (physical and logical) Nothing is static!
- 4. The relationship between individual networks Perfect“Tier1”routing Prettygoodrouting(withmanypeers;onetransit) Classicbackbonewithtwotransits(wellbalanced)
- 5. Reality … The Internet is complex (but works!) “ISP” “Transit Provider” “Transit Provider” “Transit Provider” “Transit Provider” “Transit Provider” “Transit Provider” Thisismessy! “Transit Provider” “Transit Provider”
- 6. Reality … The Internet is complex (but works!)
- 7. An introduction to peering
- 8. An introduction to peering Internet Exchange Points (IXPs) are a good idea Peering is a good idea A countries self-reliance is a good idea Critical services (DNS, NTP, etc) are a good idea The Internet is not going away; in fact it’s growing Peering helps the Internet grow
- 9. Peering/Transit - defining the terms Peering The exchange of traffic between two networks Only for traffic associated with the network, or its customers Can be via an IXP (or NAP) or a private interconnect Subject to a bilateral or unilateral contract Either zero-cost, charged-for service or as part of other contracts Majority of negotiations done on a bilateral basis Transit A network providing another network with a full connection to the Internet Sometimes provided as a “default route” or as full routes Partial Transit A network providing access to only some of the global Internet Sometimes sold as access to a specific provider or destination
- 10. Is the Internet growing? Internet growth is continuously happening More growth within IPv6 vs. IPv4; which makes sense More networks joining the global Internet routing tables each-and-every day This is just a measure of destinations on the Internet It’s not an exact measure of peering or transit relationships http://bgp.he.net/report/prefixes Network PrefixesASNs ~3,300 new networks in IPv4 table per year ~1,400 new networks in IPv6 table per year IPv4 IPv6
- 11. Who are the big Internet peering players? This is a measure of BGP adjacencies Customers are adjacencies Peers are adjacencies A high adjacencies means there’s a short path Short as in “AS PATH” vs. physical path http://bgp.he.net/report/peers IPv4 IPv6
- 13. The creation of IXPs (Internet Peering Points) IXPs provide a common facility to interconnect networks Normally defined as an IXP when three or more networks interconnect IXPs provide a common facility for traffic to flow between networks All networks are considered “equal” Peering is not transit Peering can be multilateral or bilateral IXPs can be membership based, commercial or informal IXPs provide a location for additional services
- 14. The creation of IXPs (Internet Peering Points) Initial idea was very simple (Early 1993) NSF created four “NAPs” (Network Access Points) Opened the Internet to commercial use Additional “NAP” sites (CIX & MAE-West in California) Peering Points and BGP go together perfectly City Name Pennsauken, NJ Sprint NAP San Francisco, CA PacBell NAP Chicago, IL Ameritech AADS NAP Vienna, VA MFS MAE-East
- 15. IXPs (Internet peering points) globally Tokyo JPNAP JPIX BBIX EQUINIX Los Angeles EQUINIX ANY-2 LAIIX Seattle SIX EQUINIX San Jose EQUINIX ANY-2 AMES São Paulo NAP do Brasil PTTMetro Buenos Aires CABASE Miami EQUINIX NOTA Virginia area EQUINIX ANY2 Atlanta TIE TELX EQUINIX Chicago EQUINIX Chicago IX Toronto EQUINIX TORIX Dublin INEX NJ & NY EQUINIX TIE TELX NYIIX BigApe London LINX LONAP Paris EQUINIX FRANCEIX SFINX FreeIXBrussels BNIX FreeBiXAmsterdam AMSIX NL-IX Oslo NIX Boston MXP Johannesburg NAP Africa JINX Nairobi KIXP Singapore EQUINIX SGIX Auckland APE V6IX Beijing National IX Lisbon GigaPix Wellington WIX Sydney EQUINIX PIPE Santiago NAP Chile Madrid ESPANIX Geneva CERN Moscow MSK-IX Frankfurt DE-CIX KleyReX Stockholm NetNod STHIX SOL-IX Jakarta IIX Zurich SwissIX EQUINIX Prague NIX CZ Vienna VIX Milan MIX MINAP Hong Kong HKIX EQUINIX Seoul KIDC KINX Taipei TWIX Mumbai NIXI New Delhi NIXI Kuala Lumpur MyIX Dallas EQUINIX TIE TELX Guangzhou National IX Shanghai National IX Cairo CAIX MajorIXs/NAPsrepresented;plentymoreexist
- 16. Why peering helps grow the local IP market Two points of view – each can work together Peering improves traffic flows and reduces costs Traffic flowing with lower latency (less hops?), hence is more efficient Networks that peer can reduce transit expenditure Networks that peer see local content or local eyeballs easier Peering reduces transit revenue from major players Reduced revenue is a short term effect (but it’s there) As customer experience improves; network dependence grows Peering never replaces 100% of transit needs
- 18. Acknowledgment for Hurricane Electric I would like to thank my previous employer (Hurricane Electric) for all the work presented here. The public data is available at http://bgp.he.net/ The internal data (mainly the country graphs) is not available.
- 19. Why is BGP such a powerful protocol? The IETF specified a protocol (BGP4) that could handle: Massive routing tables CIDR routing (the ability to specify an IP network address plus a network size IPv4 & IPv6 Rules for routing internally within a network vs. routing to an external network Much more BGP in real-life is used by every network on the Internet Every destination on the globe exists within the BGP global routing tables Everything is public, visible, exposed, recorded
- 20. http://bgp.he.net/ – Route propagation graphs Peers or transits that see routes ASNoriginatingroutes Routesseedownstreamofpeers Select tab for more information Transit provider Transit provider
- 21. Example listings http://bgp.he.net/country/BH A listing of all active ASNs within a country http://bgp.he.net/AS5416#_peers Alistofall“peers”ofanetworks A listing for a specific ASN
- 22. Looking at specific networks BancoNacionalde CostaRica(BNCR) TherestoftheInternet(onlymajornetworksshown) One transit provider Note: Transits for the transit
- 23. Looking at specific networks LibertyBroadcasting Network TherestoftheInternet(onlymajornetworksshown) Transit provider Transit provider
- 25. The Democratic Republic of the Congo 2013 2014
- 27. Seychelles
- 28. Lebanon
- 29. Country by country diagrams Jamaica Turks and Caicos Islands Chile Azerbaijan Gabon Sierra Leone Barbados
- 31. Kenya
- 32. Belgium
- 33. France – 1,125 active ASNs (254 with v6)
- 34. Germany
- 35. United Kingdom (Great Britain)
- 37. Regional interconnections visualized Russia, Ukraine and surrounding countries
- 39. Summary
- 40. Q&A Contact: Martin J. Levy Network Strategy Cloudflare San Francisco, CA, USA & London, UK http://cloudflare.com/ martin at cloudflare dot com +1 (408) 499 3801 ?
- 41. Additional slides - Peering Agreement review
- 42. Peering Agreement – The review What follows is a sentence-by-sentence review of the Comcast Settlement-Free Interconnect (SFI) Policy Can be found at: http://www.comcast.com/peering/ Text dated Jan/2008 June/2009 July/2011 Oct/2013 Anything with strikethrough is from an earlier version This peering language can change at any time Not actually called “peering” but “SFI”
- 43. Sample peering agreement (public information) Comcast Communications, Inc., Comcast Cable Communications Management, LLC, through its operating subsidiaries ("Comcast"), seeks to interconnect its IP network with other Internet backbone providers on a settlement-free basis when such interconnection provides tangible benefits to Comcast and its customers. For purposes of this SFI policy, an Internet Network must be a single Autonomous System (“ASN”). The criteria involving traffic measurement below applies to IPv4 IPv4 and IPv6 and must be met by the Applicant for three consecutive months. To apply for IPv4 or IPv6 settlement-free interconnection with Comcast an email must be sent to peering@comcast.com containing evidence of meeting all criteria listed in the policy. Applicants will be responded to within a reasonable timeframe to discuss their request. Networks that do not meet the requirements of the settlement-free interconnection policy, but remain interested in Comcast dedicated internet access services may also address their requests to peering@comcast.com. In this case the subject line of the e-mail should indicate a need for sales assistance. The body of the message should include as many details as possible, such as location and telephone number, to expedite the redirect for assistance.
- 44. Sample peering agreement (public information) • IPv4 IPv4 and IPv6 Settlement-Free Interconnection Requirements for Comcast AS7922 • Applicant must operate a US-wide IP backbone whose links are primarily 10 Gbps or greater. • Applicant must meet Comcast at a minimum of four mutually agreeable geographically diverse points in the US. Interconnection points must include at least one city on the US east coast, one in the central region, and one on the US west coast, and must currently be chosen from Comcast peering points in the following list of metropolitan areas: New York City/Newark NJ, Ashburn, Atlanta, Miami, Chicago, Denver, Dallas, Los Angeles, Palo Alto/San Jose, and Seattle. • Applicant’s traffic to/from the Comcast network must be on-net only and must amount to at least 3 Gbps peak 7 Gpbs peak 10 Gbps average 20 Gbps average in the dominant direction. Interconnection bandwidth must be at least 1 Gbps 10 Gbps at each interconnection point. • A network (ASN) that is a customer of a Comcast network for any dedicated IP services may not simultaneously be a settlement-free network peer. • Applicant must have a professionally managed 24x7 NOC and agree to repair or otherwise remedy any problems within a reasonable timeframe. Applicant must also agree to actively cooperate to resolve security incidents, denial of service attacks, and other operational problems.
- 45. Sample peering agreement (public information) • Applicant must maintain responsive abuse contacts for reporting and dealing with UCE (Unsolicited Commercial Email), technical contact information for capacity planning and provisioning and administrative contacts for all legal notices. • Applicant must agree to participate in joint capacity reviews at pre-set intervals and work towards timely augments as identified. • Applicant must maintain a traffic scale between its network and Comcast that enables a general balance of inbound versus outbound traffic. The end-to-end costs of network cost burden for carrying traffic between networks shall be similar to justify SFI. • Applicant must abide by the following routing policy: • Applicant must use the same peering AS at each US interconnection point and must announce a consistent set of routes at each point, unless otherwise mutually agreed. • No transit or third party routes are to be announced; all routes exchanged must be Applicant's and Applicant's customers' routes. • Applicant must filter route announcements from their customers by prefix. • Neither party shall abuse the SFI network peering relationship by engaging in activities such as, but not limited to: pointing a default route at the other or otherwise forwarding traffic for destinations not explicitly advertised, resetting next-hop, selling or giving next-hop to others.
- 46. Sample peering agreement (public information) • Applicant should be willing to enter into an NDA before formal discussions begin. • Applicant should be advised that the SFI processes will start with a 90 day trial. On successful completion of that trial, a formal interconnect agreement will be processed. This agreement will renew annually, subject to the then current SFI Policy. During the year if there is a violation of the policy, the agreement and interconnections may be terminated upon written notice to the contacts specified in the agreement. • Applicant shall not be permitted to offer or sell any IP transit services providing only AS7922. • Applicant must be financially stable. • Comcast requires that Applicants seeking SFI in the United States agree to provide reciprocal SFI arrangement with Comcast in the Applicant’s home market.
- 47. Sample peering agreement (public information) • Ongoing SFI Reviews with Comcast AS7922 • Meeting the SFI network peering guidelines set forth herein is not a guarantee that an SFI network peering relationship with Comcast will be established. Comcast shall evaluate a number of business factors and reserves the right not to enter into a SFI network peering agreement with an otherwise qualified applicant. Applicants may apply for settlement-free interconnection once every six (6) months. • Existing SFI network peers of Comcast will have their status reviewed periodically to ensure that joint capacity planning intervals are sufficient for growth, contacts are refreshed for operations purposes, and all criteria continue to be met. • Comcast reserves the right to terminate SFI network peering, upon a notice period as determined by the parties' agreement, with peers who do not meet the criteria described above. • Periodic review of the policies contained here will be conducted to ensure that the criteria and eligibility requirements are consistent with Comcast's business needs. Comcast may modify this SFI policy at any time.
- 48. Additional slides about OECD report
- 49. The OECD report OECD An International Treaty Organization Authored 2012 edition by: Dennis Weller – Navigant Economics Bill Woodcock – PCH Published October 2012 Part of a series, published every five years Audience Policy Makers, Regulators, Lawmakers, Economists, etc Atotalof99pages,108references.Plentyof Internetpeeringstats sights.org/2012/10/22/internet-traffic-exchange-2-billion-users-and-its-done-on-a-h Weller, D. and B. Woodcock (2012), “Internet Traffic Exchange: Market Developments and Policy Challenges”, OECD Digital Economy Papers, No. 207, OECD Publishing. http://dx.doi.org/10.1787/5k918gpt130q-en
- 50. The OECD report (key takeaways - globally) High handshake % for peering agreements Ratio of traffic/ecosystem cost vs. voice cost Peering and IXs work (well obviously) Very good results with little regulation
- 51. The reports peering numbers (#1) Key peering numbers: “… this report gathered responses from 4,331 ISPs … representing 86% of … in 96 countries” “… survey of 142,000 peering agreements …” “… 99.5% of interconnection agreements … without a written contract” “… less than 1% of a full mesh” “market participants will self-organize efficient Internet exchange points”
- 52. The reports peering numbers (#2) Performance: “… traffic grow at an average rate of about 50% per year” Costs: “… rates for Internet transit have declined dramatically over the last fifteen years …” “… the rate of change has leveled off in the most recent period …” “… transit … five orders of magnitude lower than typical voice”
- 53. The real audience – legal and policy Every market exists within a legal and policy framework established by government. Markets for network communications in OECD countries have benefitted from liberalisation policies which have sought to open markets and promote competition. These have included specific measures to ensure availability of leased lines and access to rights of way, for example. The success of the Internet IP market for traffic exchange would not be possible without this broad policy framework of liberalisation. Indeed, in regions where the development of Internet IP traffic exchange has been less satisfactory, the cause has generally been a lack of sufficient liberalisation within the country or region, rather than a lack of performance by the global Internet market as a whole. Legal and Policy Frameworks
- 54. Challenges for the future – four points Internet Architecture IPv4 to IPv6 Technical Innovation Speed of links need to keep up with Internet growth (10GbE to 100GbE links) Optoelectronic physics research & Internet exponential growth Internet model of traffic exchange Tromboning, peering (bilateral & multilateral), CDNs … evolution of relationships … Policy … challenges for policy makers … “bright line” between TDM world and IP world
- 55. Policy (#1) The “bright line” proposal TDM mainly lives within a regulatory framework IP traffic mainly lives without regulation VoIP (this was not reviewed – but there’s good text there) TDM reform Several OECD countries have undertaken reform
- 56. Policy (#2) Disagreements in the IP peering world “For a commercial agreement to be voluntary, parties to it must be able to decline or terminate it.” Peering disputes have been fought-out in the press and in front of potential regulators “there must be some limit to the amount of disruption a private dispute can be allowed to case” Pricing model “One allegation is that, when CDNs establish peering arrangements with local access networks, they deprive them of transit revenue. This may be true, …” Go read: A viable future model for the Internet, AT Kearney, 2011 www.atkearney.com/index.php/Publications/a-viable- future-model-for-the-internet.html Traffic exchange and network neutrality Growth of CDNs & commercial agreements & potential exclusions in the marketplace 2005 – Cogent & Level3 – fought in the press 2010 – Level3 & Comcast – all solved commercially (vs. legal or regulatory)
- 57. Regional survey (#1) “The state of development of the Internet economy differs greatly by region” “In nearly every region of the world, regulatory decisions constrain Internet development far more than technology, geography, economy, or demand. These regulatory decisions are rarely intended to benefit Internet users; they are nearly always defined by other communities of interest, at the expense of Internet users.” Differences exist regionally Africa - Latin America - Caribbean - Middle East - Europe - Canada & US - Pacific - Australia & NZ - South Asia - East Asia Success stories exist regionally Egypt, Nigeria, Ghana, Kenya & Tanzania. Brazil (CGI) & Argentina (CABASE) “Europe has more IXPs than any other region, and seven of the top ten exchanges both by number of participants and volume …” Less than successful story India and NIXI – “… current domestic exchange capacity … less than 10% of the needs of a single national broadband provider”
- 58. Regional survey (#2) Less than successful stories (an example) - Mexico “Mexico continues to lag, being by far the largest nation in the world, and the only OECD member country, to continue without any domestic Internet exchange capacity” “This situation may be on the brink of reform, as COFETEL, the Mexican regulator, has opened access to competitive long-haul circuits, has licensed a second national carrier, and is investigating the establishment of an IXP.” Source: Telegeography Figure 6: Retail price of enterprise Internet transit, mbps/month, in USD
- 59. The major takeaway Lack of Regulation has helped the Internet
Editor's Notes
- #41: IPv6