SlideShare a Scribd company logo

1-Isp-Network-Design-1

Justin Knight, profile picture
Justin Knight

The document discusses ISP network design. It covers topics such as point of presence (PoP) topologies and modules, backbone design including routing protocols and link types, and placement of ISP services like DNS. For DNS, it recommends having a primary server in a secure location, secondary servers for redundancy in diverse locations, and caching servers in each PoP for fast lookups. The document provides examples of effective DNS deployment with secondary servers around the world.

Education
1 of 104
Downloaded 12 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
Most read
48
49
50
51
52
53
54
55
Most read
56
57
58
59
60
Most read
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
ISP Network Design ISP Training Workshops 1
ISP Network Design  PoP Topologies and Design  Backbone Design  ISP Systems Design  Addressing  Routing Protocols  Security  Out of Band Management  Operational Considerations 2
Point of Presence Topologies 3
PoP Topologies  Core routers – high speed trunk connections  Distribution routers and Access routers – high port density  Border routers – connections to other providers  Service routers – hosting and servers  Some functions might be handled by a single router 4
PoP Design  Modular Design  Aggregation Services separated according to  connection speed  customer service  contention ratio  security considerations 5
Modular PoP Design 6 Backbone link to another PoP Backbone link to another PoP Leased line customer aggregation layer for leased line circuit delivery Channelised circuits Network Operations Centre Consumer DIal Access Network Core Consumer cable, xDSL and wireless Access for MetroE circuit delivery GigE fibre trunks MetroE customer aggregation layer ISP Services (DNS, Mail, News, FTP, WWW) Hosted Services & Datacentre Other ISPs Web Cache
Modular Routing Protocol Design Smaller ISPs  Modular IGP implementation  IGP “area” per PoP  Core routers in backbone area (Area 0/L2)  Aggregation/summarisation where possible into the core  Modular iBGP implementation  BGP route reflector cluster per module  Core routers are the route-reflectors  Remaining routers are clients & peer with route-reflectors only 7
Modular Routing Protocol Design Larger ISPs  Modular IGP implementation  IGP “area” per module (but avoid overloading core routers)  Core routers in backbone area (Area 0/L2)  Aggregation/summarisation where possible into the core  Modular iBGP implementation  BGP route reflector cluster per module  Dedicated route-reflectors adjacent to core routers  Clients peer with route-reflectors only 8
Point of Presence Design 9
PoP Modules  Low Speed customer connections  PSTN/ISDN dialup  Low bandwidth needs  Low revenue, large numbers  Leased line customer connections  E1/T1 speed range  Delivery over channelised media  Medium bandwidth needs  Medium revenue, medium numbers 10
PoP Modules  Broad Band customer connections  xDSL, Cable and Wireless  High bandwidth needs  Low revenue, large numbers  MetroE & Highband customer connections  Trunk onto GigE or 10GigE of 10Mbps and higher  Channelised OC3/12 delivery of E3/T3 and higher  High bandwidth needs  High revenue, low numbers 11
PoP Modules  PoP Core  Two dedicated routers  High Speed interconnect  Backbone Links ONLY  Do not touch them!  Border Network  Dedicated border router to other ISPs  The ISP’s “front” door  Transparent web caching?  Two in backbone is minimum guarantee for redundancy 12
PoP Modules  ISP Services  DNS (cache, secondary)  News (still relevant?)  Mail (POP3, Relay, Anti-virus/anti-spam)  WWW (server, proxy, cache)  Hosted Services/DataCentres  Virtual Web, WWW (server, proxy, cache)  Information/Content Services  Electronic Commerce 13
PoP Modules  Network Operations Centre  Consider primary and backup locations  Network monitoring  Statistics and log gathering  Direct but secure access  Out of Band Management Network  The ISP Network “Safety Belt” 14
Low Speed Access Module 15 To Core Routers Primary Rate T1/E1 PSTN lines to modem bank PSTN lines to built-in modems AS5400 2811 2800/3800 TACACS+/Radius proxy, DNS resolver, Content Web Cache Access Network Gateway Routers
Medium Speed Access Module 16 To Core Routers Channelised T1/E1 64K and nx64K circuits Mixture of channelised T1/E1, 56/64K and nx64K circuits 3800/7206/7600
High Speed Access Module 17 To Core Routers Metro Ethernet Channelised T3/E3 Channelised OC3/OC12 7200/7600/ASR1000/ASR9000
Broad Band Access Module 18 To Core Routers Telephone Network The cable system 6400 SSG, DHCP, TACACS+ or Radius Servers/Proxies, DNS resolver, Content Web Cache Access Network Gateway Routers uBR7246 61xx IP, ATM
ISP Services Module 19 DNS cache DNS secondary POP3 Mail Relay NEWS To core routers WWW cache Service Network Gateway Routers
Hosted Services Module 20 Customer 7 Customer 3 Customer 4 Customer 5 Customer 6 To core routers Hosted Network Gateway Routers Customer 2 Customer 1
Border Module 21 To core routers Network Border Routers To local IXP - NB - no default route + local AS routing table only ISP1 ISP2
NOC Module 22 Primary DNS To core routers Hosted Network Gateway Routers SYSLOG server TACACS+ server Network Operations Centre Staff Out of Band Management Network 2811/32async NetFlow Analyser Firewall Billing, Database and Accounting Systems Corporate LAN Critical Services Module
Out of Band Network 23 Out of Band Management Network 2811/32async To the NOC Out of Band Ethernet NetFlow Collector NetFlow enabled routers Router consoles
Backbone Network Design 24
Backbone Design  Routed Backbone  Switched Backbone  Virtually obsolete  Point-to-point circuits  nx64K, T1/E1, T3/E3, OC3, OC12, GigE, OC48, 10GigE, OC192, OC768  ATM/Frame Relay service from telco  T3, OC3, OC12,… delivery  Easily upgradeable bandwidth (CIR)  Almost vanished in availability now 25
Distributed Network Design  PoP design “standardised”  operational scalability and simplicity  ISP essential services distributed around backbone  NOC and “backup” NOC  Redundant backbone links 26
Distributed Network Design 27 POP One POP Two POP Three Customer connections Customer connections Customer connections External connections External connections Operations Centre Backup Operations Centre ISP Services ISP Services ISP Services
Backbone Links  ATM/Frame Relay  Virtually disappeared due to overhead, extra equipment, and shared with other customers of the telco  MPLS has replaced ATM & FR as the telco favourite  Leased Line/Circuit  Most popular with backbone providers  IP over Optics and Metro Ethernet very common in many parts of the world 28
Long Distance Backbone Links  Tend to cost more  Plan for the future (at least two years ahead) but stay in budget  Unplanned “emergency” upgrades can be disruptive without redundancy  Allow sufficient capacity on alternative paths for failure situations  Sufficient can be 20% to 50%  Some businesses choose 0% – meaning they have no spare capacity at all!! 29
Long Distance Links 30 POP One POP Two POP Three Long distance link Alternative/Backup Path
Metropolitan Area Backbone Links  Tend to be cheaper  Circuit concentration  Choose from multiple suppliers  Think big  More redundancy  Less impact of upgrades  Less impact of failures 31
Metropolitan Area Backbone Links 32 POP One POP Two POP Three Metropolitan Links Metropolitan Links Traditional Point to Point Links
ISP Services DNS, Mail, News Design and Placement 33
ISP Services  Most ISP services such as DNS, Mail, etc are easily deliverable on low budget hardware platforms  Single Rack Unit in height (1RU)  Dual processor is “default” now  RAM is very cheap (may as well use 2Gbytes or more)  Hard drives are very cheap (SCSI more reliable)  Unix like operating systems (FreeBSD, Debian, Ubuntu, CentOS) are very common  In addition to commercial operating systems such as Solaris, RedHat Enterprise Linux &c  Minimal overhead, minimal OS install, plus the service required 34
ISP Services: DNS  Domain Name System  Provides name and address resolution  Servers need to be differentiated, properly located and specified  Primary nameserver  Secondary nameserver  Caching nameserver – resolver 35
ISP Services: DNS  Primary nameserver  Holds ISP zone files  Forward zone (list of name to address mappings) for all ISP’s and any customer zones  Reverse zone (list of address to name mappings) for all ISP’s address space  Hardware & OS: easily satisfied by simple specification  Located in secure part of net, e.g. NOC LAN  Usually run as “hidden master” – secondary nameservers are the official listed nameservers 36
ISP Services: DNS  Secondary nameserver  Holds copies of ISP zone files  At least two are required, more is better  Hardware & OS: easily satisfied by simple specification  Strongly recommended to be geographically separate from each other and the primary DNS  At different PoPs  On a different continent e.g. via services offered by ISC, PCH and others  At another ISP 37
ISP Services: Secondary DNS Example 38 $ dig apnic.net ns ;; ANSWER SECTION: apnic.net. 10800 NS ns1.apnic.net. apnic.net. 10800 NS ns3.apnic.net. apnic.net. 10800 NS ns4.apnic.net. apnic.net. 10800 NS ns5.apnic.com. apnic.net. 10800 NS cumin.apnic.net. apnic.net. 10800 NS ns-sec.ripe.net. apnic.net. 10800 NS tinnie.arin.net. apnic.net. 10800 NS tinnie.apnic.net. ;; ADDITIONAL SECTION: ns1.apnic.net. 3600 A 202.12.29.25 ns3.apnic.net. 3600 A 202.12.28.131 ns4.apnic.net. 3600 A 202.12.31.140 ns5.apnic.com. 10800 A 203.119.43.200 cumin.apnic.net. 3600 A 202.12.29.59 tinnie.apnic.net. 3600 A 202.12.29.60 ns-sec.ripe.net. 113685 A 193.0.0.196 tinnie.arin.net. 10800 A 199.212.0.53 Tokyo Hong Kong Washington Brisbane Brisbane Amsterdam Washington
ISP Services: Secondary DNS Example  apnic.net zone  Primary DNS in Brisbane (ns1.apnic.net)  Secondary DNS run all over the world by APNIC:  Brisbane  Hong Kong  Tokyo  Washington  Zone secondaried by  RIPE NCC in Amsterdam  ARIN in Washington  Geographical and service provider redundancy – this is the perfect example! 39
ISP Services: DNS  Caching nameserver  This is the resolver – it is the DNS cache  Your customers use this as resolver, NEVER your primary or secondary DNS  Provides very fast lookups  Does NOT secondary any zones  One, or preferably two per PoP (redundancy)  Hardware & OS: easily satisfied by simple specification 40
ISP Services: Caching Nameserver  DIAL users automatically given the IP addresses of DNS caches when they dial in 41 To Core Routers DIAL network Web Cache DNS Cache DNS Cache Radius proxy Switch redundancy Router redundancy DNS Cache redundancy
ISP Services: Anycasting the Caching Nameserver  One trick of the trade  assign two unique IP addresses to be used for the two DNS resolver systems  use these two IP addresses in every PoP  route the two /32s across your backbone  even if the two resolver systems in the local PoP are down, the IGP will ensure that the next nearest resolvers will be reachable  Known as IP Anycast 42 Geek Alert
ISP Services: DNS  Efficient and resilient design  Primary DNS – keep it secure  Secondary DNS – geographical and provider redundancy  Don’t ever put them on the same LAN, switched or otherwise  Don’t put them in the same PoP  Caching DNS – one or two per PoP  Reduces DNS traffic across backbone  More efficient, spreads the load 43
ISP Services: DNS  Software  Make sure that the BIND distribution on the Unix system is up to date  The vendor’s distribution is rarely current  Pay attention to bug reports, security issues  Reboot the DNS cache on a regular (e.g. monthly) basis  Clears out the cache  Releases any lost RAM  Accepted good practice by system administrators 44
ISP Services: DNS  Implementation  Put all your hosts, point-to-point links and loopbacks into the DNS  Under your ISP’s domain name  Use sensible/meaningful names  Put all your hosts, point-to-point links and loopbacks into the REVERSE DNS also  Don’t forget about in-addr.arpa and ip6.arpa – many ISPs do  Some systems demand forward/reverse DNS mapping before allowing access 45
ISP Services: Mail  Must have at least two mail hosts (MX records) for all supported domains  Geographical separation helps  Dedicated POP3 server  Consumers/mobile users get mail from here  SMTP gateway dedicated to that function  Consumers/mobile users send mail via here  Mail relay open to CUSTOMERS only!  Don’t let outside world use your mail relay  Block port 25 outbound for all customers  Insist that outbound e-mail goes through SMTP relay  SMTP relay does virus (ClamAV) and spam (Spamassassin) filtering 46
ISP Services: Mail Configuration 47 smtp.isp.net Customer mail relay Incoming mail from customer mail.isp.net ISP Mail Gateway Incoming mail from Internet pop3.isp.net Customer POP3/IMAP server Mail pulled by customer client Mail out to the Internet SpamAssassin ClamAV SpamAssassin ClamAV
ISP Services: Mail Example  cisco.com mail (MX records)  primary MX are 6 systems in San Jose  Three backup MXes in RTP, Amsterdam and Sydney  backup MX only used if primary unavailable 48 $ dig cisco.com mx ;; ANSWER SECTION: cisco.com. 86400 MX 10 sj-inbound-a.cisco.com. cisco.com. 86400 MX 10 sj-inbound-b.cisco.com. cisco.com. 86400 MX 10 sj-inbound-c.cisco.com. cisco.com. 86400 MX 10 sj-inbound-d.cisco.com. cisco.com. 86400 MX 10 sj-inbound-e.cisco.com. cisco.com. 86400 MX 10 sj-inbound-f.cisco.com. cisco.com. 86400 MX 15 rtp-mx-01.cisco.com. cisco.com. 86400 MX 20 ams-inbound-a.cisco.com. cisco.com. 86400 MX 25 syd-inbound-a.cisco.com.
ISP Services: Mail  Software  Make sure that the MAIL and POP3 distributions on the Unix system are up to date  The vendor distributions are rarely current  Pay attention to bug reports, security issues, unsolicited junk mail complaints 49 IMPORTANT: Do NOT allow non-customers to use your mail system as a relay
ISP Services: News  News servers provide a Usenet news feed to customers  Distributed design required  Incoming newsfeed to one large server  Distributed to feed servers in each PoP  Feed servers provide news feed to customers  Outgoing news goes to another server  Separate reading news system  Separate posting news system 50
ISP Services: News System Placement 51 POP One POP Two POP Three Customer connections Customer connections Customer connections External connections External connections News Collector News Feeder News Feeder News Feeder News Distributor
ISP Services: News System Placement 52 POP One POP Two POP Three Customer connections Customer connections Customer connections External connections External connections News Collector News Feeder News Feeder News Feeder News Distributor
ISP Services: News  Software  Make sure that the Internet News distribution on the Unix system is up to date  The vendor distributions are rarely current  Pay attention to bug reports, security issues, unsolicited junk posting complaints 53 IMPORTANT: Do NOT allow non-customers to use your news system for posting messages
Addressing 54
Where to get IP addresses and AS numbers  Your upstream ISP  Africa  AfriNIC – http://www.afrinic.net  Asia and the Pacific  APNIC – http://www.apnic.net  North America  ARIN – http://www.arin.net  Latin America and the Caribbean  LACNIC – http://www.lacnic.net  Europe and Middle East  RIPE NCC – http://www.ripe.net/info/ncc 55
Internet Registry Regions 56
Getting IP address space  Take part of upstream ISP’s PA space or  Become a member of your Regional Internet Registry and get your own allocation  Require a plan for a year ahead  General policies are outlined in RFC2050, more specific details are on the individual RIR website  There is no more IPv4 address space at IANA  Most RIRs are now entering their “final /8” IPv4 delegation policies  Limited IPv4 available  IPv6 allocations are simple to get in most RIR regions 57
What about RFC1918 addressing?  RFC1918 defines IP addresses reserved for private Internets  Not to be used on Internet backbones  http://www.ietf.org/rfc/rfc1918.txt  Commonly used within end-user networks  NAT used to translate from private internal to public external addressing  Allows the end-user network to migrate ISPs without a major internal renumbering exercise  Most ISPs filter RFC1918 addressing at their network edge  http://www.cymru.com/Documents/bogon- list.html 58
What about RFC1918 addressing?  List of well known problems with this approach for an SP backbone:  Breaks Path MTU Discovery  Potential conflicts with usage of private addressing inside customer networks  Security through obscurity does not provide security  Troubleshooting outside the local network becomes very hard  Router interface addresses are only locally visible  Internet becomes invisible from the router  Troubleshooting of connectivity issues on an Internet scale becomes impossible  Traceroutes and pings provide no information  No distinction between “network invisible” and “network broken”  Increases operational complexity of the network infrastructure and routing configuration 59
Private versus Globally Routable IP Addressing  Infrastructure Security: not improved by using private addressing  Still can be attacked from inside, or from customers, or by reflection techniques from the outside  Troubleshooting: made an order of magnitude harder  No Internet view from routers  Other ISPs cannot distinguish between down and broken  Performance: PMTUD breakage  Summary:  ALWAYS use globally routable IP addressing for ISP Infrastructure 60
Addressing Plans – ISP Infrastructure  Address block for router loop-back interfaces  Address block for infrastructure  Per PoP or whole backbone  Summarise between sites if it makes sense  Allocate according to genuine requirements, not historic classful boundaries  Similar allocation policies should be used for IPv6 as well  ISPs just get a substantially larger block (relatively) so assignments within the backbone are easier to make 61
Addressing Plans – Customer  Customers are assigned address space according to need  Should not be reserved or assigned on a per PoP basis  ISP iBGP carries customer nets  Aggregation not required and usually not desirable 62
Addressing Plans – ISP Infrastructure  Phase One 63 223.10.0.0/21 Customer assignments Infrastructure Loopbacks /24 223.10.6.255 223.10.0.1 223.10.0.0/20 Original assignments New Assignments /24 /24 223.10.0.1 223.10.5.255 223.10.15.255 Phase Two
Addressing Plans Planning  Registries will usually allocate the next block to be contiguous with the first allocation  Minimum allocation could be /21  Very likely that subsequent allocation will make this up to a /20  So plan accordingly 64
Addressing Plans (contd)  Document infrastructure allocation  Eases operation, debugging and management  Document customer allocation  Contained in iBGP  Eases operation, debugging and management  Submit network object to RIR Database 65
Routing Protocols 66
Routing Protocols  IGP – Interior Gateway Protocol  carries infrastructure addresses, point-to-point links  examples are OSPF, ISIS,...  EGP – Exterior Gateway Protocol  carries customer prefixes and Internet routes  current EGP is BGP version 4  No connection between IGP and EGP 67
Why Do We Need an IGP?  ISP backbone scaling  Hierarchy  Modular infrastructure construction  Limiting scope of failure  Healing of infrastructure faults using dynamic routing with fast convergence 68
Why Do We Need an EGP?  Scaling to large network  Hierarchy  Limit scope of failure  Policy  Control reachability to prefixes  Merge separate organizations  Connect multiple IGPs 69
Interior versus Exterior Routing Protocols  Interior  Automatic neighbour discovery  Generally trust your IGP routers  Prefixes go to all IGP routers  Binds routers in one AS together  Exterior  Specifically configured peers  Connecting with outside networks  Set administrative boundaries  Binds AS’s together 70
Interior versus Exterior Routing Protocols  Interior  Carries ISP infrastructure addresses only  ISPs aim to keep the IGP small for efficiency and scalability  Exterior  Carries customer prefixes  Carries Internet prefixes  EGPs are independent of ISP network topology 71
Hierarchy of Routing Protocols 72 BGP4 BGP4 and OSPF/ISIS Other ISPs Customers IXP Static/BGP4 BGP4
Routing Protocols: Choosing an IGP  Review the “OSPF vs ISIS” presentation:  OSPF and ISIS have very similar properties  ISP usually chooses between OSPF and ISIS  Choose which is appropriate for your operators’ experience  In most vendor releases, both OSPF and ISIS have sufficient “nerd knobs” to tweak the IGP’s behaviour  OSPF runs on IP  ISIS runs on infrastructure, alongside IP 73
Routing Protocols: IGP Recommendations  Keep the IGP routing table as small as possible  If you can count the routers and the point to point links in the backbone, that total is the number of IGP entries you should see  IGP details:  Should only have router loopbacks, backbone WAN point-to-point link addresses, and network addresses of any LANs having an IGP running on them  Strongly recommended to use inter-router authentication  Use inter-area summarisation if possible 74
Routing Protocols: More IGP recommendations  To fine tune IGP table size more, consider:  Using “ip unnumbered” on customer point-to- point links – saves carrying that /30 in IGP  (If customer point-to-point /30 is required for monitoring purposes, then put this in iBGP)  Use contiguous addresses for backbone WAN links in each area – then summarise into backbone area  Don’t summarise router loopback addresses – as iBGP needs those (for next-hop)  Use iBGP for carrying anything which does not contribute to the IGP Routing process 75
Routing Protocols: iBGP Recommendations  iBGP should carry everything which doesn’t contribute to the IGP routing process  Internet routing table  Customer assigned addresses  Customer point-to-point links  Dial network pools, passive LANs, etc 76
Routing Protocols: More iBGP Recommendations  Scalable iBGP features:  Use neighbour authentication  Use peer-groups to speed update process and for configuration efficiency  Use communities for ease of filtering  Use route-reflector hierarchy  Route reflector pair per PoP (overlaid clusters) 77
Security 78
Security  ISP Infrastructure security  ISP Network security  Security is not optional!  ISPs need to:  Protect themselves  Help protect their customers from the Internet  Protect the Internet from their customers  The following slides are general recommendations  Do more research on security before deploying any network 79
ISP Infrastructure Security  Router security  Usernames, passwords, vty filters, TACACS+  Disable telnet on vtys, only use SSH  vty filters should only allow NOC access, no external access  See IOS Essentials for the recommended practices for ISPs 80
ISP Infrastructure Security  ISP server security  Usernames, passwords, TCP wrappers, IPTABLES  Protect all servers using routers with strong filters applied  Hosted services security  Protect network from hosted servers using routers with strong filters  Protect hosted servers from Internet using routers with strong filters 81
ISP Infrastructure Security ISP Server Protection 82 DNS cache DNS secondary POP3 Mail Relay NEWS To core routers Service Network Gateway Routers Access-list examples: Allow tcp/established to all servers ICMP DNS 2ary: udp/53 and tcp/53 POP3: tcp/110 Mail Relay: tcp/25 and ISP address range only News: tcp/119 and ISP address range only DNS Cache: udp/53 Web server: tcp/80 Other necessary filters: All servers: SSH (tcp/22) from NOC LAN only Web server
ISP Infrastructure Security Hosted Server Protection 83 Access-list examples: Inbound Allow tcp/established to all servers ICMP Web server: tcp/80 SSH for customer access Any other ports for services sold to customers Outbound ICMP Allow DNS udp/53 and tcp/53 Block all access to ISP address range Server5 Server1 Server2 Server3 Server4 To core routers Service Network Gateway Routers Server6
ISP Infrastructure Security  Premises security  Locks – electronic/card key preferred  Secure access – 24x7 security arrangements  Environment control – good aircon  Staff responsibility  Password policy, strangers, temp staff  Employee exit procedures  RFC2196  (Site Security Handbook)  RFC3871  (Operational Security Requirements for Large ISP IP Network Infrastructure ) 84
ISP Network Security  Denial of Service Attacks  eg: “smurfing”  see http://www.denialinfo.com  Effective filtering  Network borders – see Cisco ISP Essentials  Customer connections – unicast RPF on ALL of them  Network operation centre  ISP corporate network – behind firewall 85
ISP Network Security Secure external access  How to provide staff access from outside  Set up ssh gateway (Unix system with ssh daemon and nothing else configured)  Provide ssh client on all staff laptops  ssh available on Unix and Windows  ssh is Secure Shell – encrypted link  How not to provide access from outside  telnet, rsh, rlogin – these are all insecure  Open host – insecure, can be compromised 86
Ingress & Egress Route Filtering Your customers should not be sending any IP packets out to the Internet with a source address other then the address you have allocated to them! 87
Out of Band Management 88
Out of Band Management  Not optional!  Allows access to network equipment in times of failure  Ensures quality of service to customers  Minimises downtime  Minimises repair time  Eases diagnostics and debugging 89
Out of Band Management  OoB Example – Access server:  modem attached to allow NOC dial in  console ports of all network equipment connected to serial ports  LAN and/or WAN link connects to network core, or via separate management link to NOC  Full remote control access under all circumstances 90
Out of Band Network 91 Ethernet to the NOC Router, switch and ISP server consoles (Optional) Out of band WAN link to other PoPs Modem – access to PSTN for out of band dialin Equipment Rack Equipment Rack
Out of Band Management  OoB Example – Statistics gathering:  Routers are NetFlow and syslog enabled  Management data is congestion/failure sensitive  Ensures management data integrity in case of failure  Full remote information under all circumstances 92
Test Laboratory 93
Test Laboratory  Designed to look like a typical PoP  Operated like a typical PoP  Used to trial new services or new software under realistic conditions  Allows discovery and fixing of potential problems before they are introduced to the network 94
Test Laboratory  Some ISPs dedicate equipment to the lab  Other ISPs “purchase ahead” so that today’s lab equipment becomes tomorrow’s PoP equipment  Other ISPs use lab equipment for “hot spares” in the event of hardware failure 95
Test Laboratory  Can’t afford a test lab?  Set aside one spare router and server to trial new services  Never ever try out new hardware, software or services on the live network  Every major ISP in the US and Europe has a test lab  It’s a serious consideration 96
Operational Considerations 97
Operational Considerations 98 Why design the world’s best network when you have not thought about what operational good practices should be implemented?
Operational Considerations Maintenance  Never work on the live network, no matter how trivial the modification may seem  Establish maintenance periods which your customers are aware of  e.g. Tuesday 4-7am, Thursday 4-7am  Never do maintenance on a Friday  Unless you want to work all weekend cleaning up  Never do maintenance on a Monday  Unless you want to work all weekend preparing 99
Operational Considerations Support  Differentiate between customer support and the Network Operations Centre  Customer support fixes customer problems  NOC deals with and fixes backbone and Internet related problems  Network Engineering team is last resort  They design the next generation network, improve the routing design, implement new services, etc  They do not and should not be doing support! 100
Operational Considerations NOC Communications  NOC should know contact details for equivalent NOCs in upstream providers and peers  Or consider joining the INOC-DBA system  Voice over IP phone system using SIP  Runs over the Internet  www.pch.net/inoc-dba for more information 101
ISP Network Design Summary 102
ISP Design Summary  KEEP IT SIMPLE & STUPID ! (KISS)  Simple is elegant is scalable  Use Redundancy, Security, and Technology to make life easier for yourself  Above all, ensure quality of service for your customers 103
ISP Network Design ISP Training Workshops 104

More Related Content

PPT
Gpon Fundamentals
mansoor_gr8
 
PDF
PON design considerations for FTTH FTTx
Sun Telecommunication (Shanghai) Co., Ltd.
 
PPT
Vlan final
Veena Rao
 
PPTX
WLAN
Mukesh Chinta
 
PPT
ngn overview , next generation network
Tushar Kumar
 
PPTX
Network Design on cisco packet tracer 6.0
Saurav Pandey
 
PPT
Public Switched Telephone Network (PSTN)
J.T.A.JONES
 
PPT
52528672 microwave-planning-and-design
fat_zeq
 
Gpon Fundamentals
mansoor_gr8
 
PON design considerations for FTTH FTTx
Sun Telecommunication (Shanghai) Co., Ltd.
 
Vlan final
Veena Rao
 
WLAN
Mukesh Chinta
 
ngn overview , next generation network
Tushar Kumar
 
Network Design on cisco packet tracer 6.0
Saurav Pandey
 
Public Switched Telephone Network (PSTN)
J.T.A.JONES
 
52528672 microwave-planning-and-design
fat_zeq
 

What's hot (20)

PPTX
Mobile Networks Overview (2G / 3G / 4G-LTE)
Hamidreza Bolhasani
 
PPTX
Basics of DWDM Technology
Pankaj Lahariya
 
PPT
FTTH
Sahil Koundal
 
PPTX
RF Planning & Optimization
Syed Abdul Basit
 
PPTX
Optical fiber Communication training ppt
huzaifa027
 
PDF
Passive infrastructure of FTTH networks: an overview
Luc De Heyn
 
PDF
Understanding olt, onu, ont and odn full
S M Tipu
 
PDF
Optical Transport Network
Anuradha Udunuwara
 
PPT
How PSTN phone works?
mahipal9
 
PPTX
2G / 3G / 4G / IMS / 5G Overview with Focus on Core Network
Hamidreza Bolhasani
 
PPTX
5G Network: Requirements, Design Principles, Architectures, and Enabling Tech...
Sayed Chhattan Shah
 
PDF
Difference between Routing & Routed Protocol
Netwax Lab
 
PDF
FTTX Basics and Network Design
OFS
 
PPT
Basics Of Minilink Microwave Networks
Atif Mahmood
 
DOCX
Wireless lan
Fʑɭ Rɘʜɱʌŋ
 
PDF
Mobile network structure
Ahmed Hussien Ali Gomaa Bebars
 
PDF
FTTH Basics
Anuradha Udunuwara
 
PDF
Carrier Ethernet - What and Why
Anuradha Udunuwara
 
PPTX
Bsnl best ppt
rajat pundir
 
PPTX
Basics of Optical Network Architecture, PON & GPON
Syed Shujat Ali
 
Mobile Networks Overview (2G / 3G / 4G-LTE)
Hamidreza Bolhasani
 
Basics of DWDM Technology
Pankaj Lahariya
 
FTTH
Sahil Koundal
 
RF Planning & Optimization
Syed Abdul Basit
 
Optical fiber Communication training ppt
huzaifa027
 
Passive infrastructure of FTTH networks: an overview
Luc De Heyn
 
Understanding olt, onu, ont and odn full
S M Tipu
 
Optical Transport Network
Anuradha Udunuwara
 
How PSTN phone works?
mahipal9
 
2G / 3G / 4G / IMS / 5G Overview with Focus on Core Network
Hamidreza Bolhasani
 
5G Network: Requirements, Design Principles, Architectures, and Enabling Tech...
Sayed Chhattan Shah
 
Difference between Routing & Routed Protocol
Netwax Lab
 
FTTX Basics and Network Design
OFS
 
Basics Of Minilink Microwave Networks
Atif Mahmood
 
Wireless lan
Fʑɭ Rɘʜɱʌŋ
 
Mobile network structure
Ahmed Hussien Ali Gomaa Bebars
 
FTTH Basics
Anuradha Udunuwara
 
Carrier Ethernet - What and Why
Anuradha Udunuwara
 
Bsnl best ppt
rajat pundir
 
Basics of Optical Network Architecture, PON & GPON
Syed Shujat Ali
 
Ad

Similar to 1-Isp-Network-Design-1 (20)

PPT
L6 6 lowpan
bimal2638
 
PPT
Wan networks
Arnold Derrick Kinney
 
PPTX
Network.pptx
SAMANTHACARDOSO13
 
PPT
Resilient Network Design Concepts Educat
SamGrandprix
 
PPT
NET6.PPT
ssuser50c54b
 
PPT
Sinnreich Henry Johnston Alan Pt 2
Carl Ford
 
PPTX
A10 presentation for LB GLB for ADC solutions
eyala75
 
PPT
Mpls vpn1
rel comm
 
PDF
Is there still room for linking narrowband PMR sites in this digital world?
Comms Connect
 
PDF
JAWDAT NGN IDNOG v1.0 (public)[COPY]
Mahadiputra S
 
PDF
Jawdat NGN IDNOG v1.0 public
Himawan Nugroho
 
PDF
03 (IDNOG01) NGN Next Generation Networks by Himawan Nugroho
Indonesia Network Operators Group
 
PPT
Chapter 02 - Wan Router
phanleson
 
PDF
Haystack Technology Overview
Haystack Technologies
 
PPT
High Performance Communication for Oracle using InfiniBand
webhostingguy
 
PPT
Chapter11
Muhammad Ahad
 
PPT
Networking Concepts Lesson 06 - Protocols - Eric Vanderburg
Eric Vanderburg
 
PPTX
Small office Home office , network setup in details
apel7
 
PPTX
16.) layer 3 (basic tcp ip routing)
Jeff Green
 
PDF
Protocols
Abdelaziz Elbaze
 
L6 6 lowpan
bimal2638
 
Wan networks
Arnold Derrick Kinney
 
Network.pptx
SAMANTHACARDOSO13
 
Resilient Network Design Concepts Educat
SamGrandprix
 
NET6.PPT
ssuser50c54b
 
Sinnreich Henry Johnston Alan Pt 2
Carl Ford
 
A10 presentation for LB GLB for ADC solutions
eyala75
 
Mpls vpn1
rel comm
 
Is there still room for linking narrowband PMR sites in this digital world?
Comms Connect
 
JAWDAT NGN IDNOG v1.0 (public)[COPY]
Mahadiputra S
 
Jawdat NGN IDNOG v1.0 public
Himawan Nugroho
 
03 (IDNOG01) NGN Next Generation Networks by Himawan Nugroho
Indonesia Network Operators Group
 
Chapter 02 - Wan Router
phanleson
 
Haystack Technology Overview
Haystack Technologies
 
High Performance Communication for Oracle using InfiniBand
webhostingguy
 
Chapter11
Muhammad Ahad
 
Networking Concepts Lesson 06 - Protocols - Eric Vanderburg
Eric Vanderburg
 
Small office Home office , network setup in details
apel7
 
16.) layer 3 (basic tcp ip routing)
Jeff Green
 
Protocols
Abdelaziz Elbaze
 
Ad

More from Justin Knight (20)

PDF
My Summer Narrative Writing For The Beginning O
Justin Knight
 
PDF
Writing Paper With Drawing Space. Online assignment writing service.
Justin Knight
 
PDF
College Admission Essay Template - College Applica
Justin Knight
 
PDF
How To Open An Essay. How To Start An Essay (With Pictures. Online assignment...
Justin Knight
 
PDF
College Application Essay Sample College Applicatio
Justin Knight
 
PDF
Top Narrative Essay Examples Mla Most Popul
Justin Knight
 
PDF
Calamo - How To Buy Best Essay Wr. Online assignment writing service.
Justin Knight
 
PDF
Career Research Paper Sample. Online assignment writing service.
Justin Knight
 
PDF
Writing The Ad Analysis Essay - YouTube. Online assignment writing service.
Justin Knight
 
PDF
ABAP FAQ S On Reports Scripts BDC Dialogs ABAP Reporting SAP TERMINOLOGY
Justin Knight
 
PDF
AI 03 Solving Problems By Searching
Justin Knight
 
PDF
ASSESSMENT AND EVALUATION IN EDUCATION
Justin Knight
 
PDF
Application Of Lean Six Sigma To Improve Service In Healthcare Facilities Man...
Justin Knight
 
PDF
A Framework For Information Security Risk Management Communication
Justin Knight
 
PDF
An Object Approach For Web Presentations
Justin Knight
 
PDF
American Ways - An Introducation To American Culture.PDF
Justin Knight
 
PDF
A Compilation Of Case Digests For Transportation Law
Justin Knight
 
PDF
A New Childhood Social Studies Curriculum For A New Generation Of Citizenship
Justin Knight
 
PDF
A Supporting Hand In Dealing With Interpersonal Conflicts The Role Of Intera...
Justin Knight
 
PDF
An Analytical Framework For Miles And Snow Typology And Dynamic Capabilities
Justin Knight
 
My Summer Narrative Writing For The Beginning O
Justin Knight
 
Writing Paper With Drawing Space. Online assignment writing service.
Justin Knight
 
College Admission Essay Template - College Applica
Justin Knight
 
How To Open An Essay. How To Start An Essay (With Pictures. Online assignment...
Justin Knight
 
College Application Essay Sample College Applicatio
Justin Knight
 
Top Narrative Essay Examples Mla Most Popul
Justin Knight
 
Calamo - How To Buy Best Essay Wr. Online assignment writing service.
Justin Knight
 
Career Research Paper Sample. Online assignment writing service.
Justin Knight
 
Writing The Ad Analysis Essay - YouTube. Online assignment writing service.
Justin Knight
 
ABAP FAQ S On Reports Scripts BDC Dialogs ABAP Reporting SAP TERMINOLOGY
Justin Knight
 
AI 03 Solving Problems By Searching
Justin Knight
 
ASSESSMENT AND EVALUATION IN EDUCATION
Justin Knight
 
Application Of Lean Six Sigma To Improve Service In Healthcare Facilities Man...
Justin Knight
 
A Framework For Information Security Risk Management Communication
Justin Knight
 
An Object Approach For Web Presentations
Justin Knight
 
American Ways - An Introducation To American Culture.PDF
Justin Knight
 
A Compilation Of Case Digests For Transportation Law
Justin Knight
 
A New Childhood Social Studies Curriculum For A New Generation Of Citizenship
Justin Knight
 
A Supporting Hand In Dealing With Interpersonal Conflicts The Role Of Intera...
Justin Knight
 
An Analytical Framework For Miles And Snow Typology And Dynamic Capabilities
Justin Knight
 

Recently uploaded (20)

PDF
Supply Chain Operations Speaking Notes -ICLT Program
labyankof
 
PDF
O5-L3 Freight Transport Ops (International) V1.pdf
labyankof
 
PDF
Basic Mud Logging Guide for educational purpose
wdandworks
 
PPTX
Pharma ospi slides which help in ospi learning
rameetkumar304
 
PDF
VCE English Exam - Section C Student Revision Booklet
jpinnuck
 
PDF
Sports Quiz easy sports quiz sports quiz
nikunj2757
 
PDF
102 student loan defaulters named and shamed – Is someone you know on the list?
Kweku Zurek
 
PPTX
1st Inaugural Professorial Lecture held on 19th February 2020 (Governance and...
kawisojames10
 
PDF
Saundersa Comprehensive Review for the NCLEX-RN Examination.pdf
sreejithcareers
 
PPTX
master seminar digital applications in india
ananyaray35
 
PPTX
Institutional Correction lecture only . . .
limvtheghins
 
PPTX
Pharmacology of Heart Failure /Pharmacotherapy of CHF
Rajshri Ghogare
 
PPTX
Introduction_to_Human_Anatomy_and_Physiology_for_B.Pharm.pptx
chetansingh379583
 
PDF
Complications of Minimal Access Surgery at WLH
mohitsuren827
 
PPTX
GDM (1) (1).pptx small presentation for students
shrawanbpkihs
 
PPTX
Microbial diseases, their pathogenesis and prophylaxis
DevlinaSengupta
 
PDF
Computing-Curriculum for Schools in Ghana
abigailanane203
 
PDF
RMMM.pdf make it easy to upload and study
sajedul2
 
PDF
FourierSeries-QuestionsWithAnswers(Part-A).pdf
Raji Murugan
 
PDF
Module 4: Burden of Disease Tutorial Slides S2 2025
Jonathan Hallett
 
Supply Chain Operations Speaking Notes -ICLT Program
labyankof
 
O5-L3 Freight Transport Ops (International) V1.pdf
labyankof
 
Basic Mud Logging Guide for educational purpose
wdandworks
 
Pharma ospi slides which help in ospi learning
rameetkumar304
 
VCE English Exam - Section C Student Revision Booklet
jpinnuck
 
Sports Quiz easy sports quiz sports quiz
nikunj2757
 
102 student loan defaulters named and shamed – Is someone you know on the list?
Kweku Zurek
 
1st Inaugural Professorial Lecture held on 19th February 2020 (Governance and...
kawisojames10
 
Saundersa Comprehensive Review for the NCLEX-RN Examination.pdf
sreejithcareers
 
master seminar digital applications in india
ananyaray35
 
Institutional Correction lecture only . . .
limvtheghins
 
Pharmacology of Heart Failure /Pharmacotherapy of CHF
Rajshri Ghogare
 
Introduction_to_Human_Anatomy_and_Physiology_for_B.Pharm.pptx
chetansingh379583
 
Complications of Minimal Access Surgery at WLH
mohitsuren827
 
GDM (1) (1).pptx small presentation for students
shrawanbpkihs
 
Microbial diseases, their pathogenesis and prophylaxis
DevlinaSengupta
 
Computing-Curriculum for Schools in Ghana
abigailanane203
 
RMMM.pdf make it easy to upload and study
sajedul2
 
FourierSeries-QuestionsWithAnswers(Part-A).pdf
Raji Murugan
 
Module 4: Burden of Disease Tutorial Slides S2 2025
Jonathan Hallett
 

1-Isp-Network-Design-1

  • 1. ISP Network Design ISP Training Workshops 1
  • 2. ISP Network Design  PoP Topologies and Design  Backbone Design  ISP Systems Design  Addressing  Routing Protocols  Security  Out of Band Management  Operational Considerations 2
  • 4. PoP Topologies  Core routers – high speed trunk connections  Distribution routers and Access routers – high port density  Border routers – connections to other providers  Service routers – hosting and servers  Some functions might be handled by a single router 4
  • 5. PoP Design  Modular Design  Aggregation Services separated according to  connection speed  customer service  contention ratio  security considerations 5
  • 6. Modular PoP Design 6 Backbone link to another PoP Backbone link to another PoP Leased line customer aggregation layer for leased line circuit delivery Channelised circuits Network Operations Centre Consumer DIal Access Network Core Consumer cable, xDSL and wireless Access for MetroE circuit delivery GigE fibre trunks MetroE customer aggregation layer ISP Services (DNS, Mail, News, FTP, WWW) Hosted Services & Datacentre Other ISPs Web Cache
  • 7. Modular Routing Protocol Design Smaller ISPs  Modular IGP implementation  IGP “area” per PoP  Core routers in backbone area (Area 0/L2)  Aggregation/summarisation where possible into the core  Modular iBGP implementation  BGP route reflector cluster per module  Core routers are the route-reflectors  Remaining routers are clients & peer with route-reflectors only 7
  • 8. Modular Routing Protocol Design Larger ISPs  Modular IGP implementation  IGP “area” per module (but avoid overloading core routers)  Core routers in backbone area (Area 0/L2)  Aggregation/summarisation where possible into the core  Modular iBGP implementation  BGP route reflector cluster per module  Dedicated route-reflectors adjacent to core routers  Clients peer with route-reflectors only 8
  • 9. Point of Presence Design 9
  • 10. PoP Modules  Low Speed customer connections  PSTN/ISDN dialup  Low bandwidth needs  Low revenue, large numbers  Leased line customer connections  E1/T1 speed range  Delivery over channelised media  Medium bandwidth needs  Medium revenue, medium numbers 10
  • 11. PoP Modules  Broad Band customer connections  xDSL, Cable and Wireless  High bandwidth needs  Low revenue, large numbers  MetroE & Highband customer connections  Trunk onto GigE or 10GigE of 10Mbps and higher  Channelised OC3/12 delivery of E3/T3 and higher  High bandwidth needs  High revenue, low numbers 11
  • 12. PoP Modules  PoP Core  Two dedicated routers  High Speed interconnect  Backbone Links ONLY  Do not touch them!  Border Network  Dedicated border router to other ISPs  The ISP’s “front” door  Transparent web caching?  Two in backbone is minimum guarantee for redundancy 12
  • 13. PoP Modules  ISP Services  DNS (cache, secondary)  News (still relevant?)  Mail (POP3, Relay, Anti-virus/anti-spam)  WWW (server, proxy, cache)  Hosted Services/DataCentres  Virtual Web, WWW (server, proxy, cache)  Information/Content Services  Electronic Commerce 13
  • 14. PoP Modules  Network Operations Centre  Consider primary and backup locations  Network monitoring  Statistics and log gathering  Direct but secure access  Out of Band Management Network  The ISP Network “Safety Belt” 14
  • 15. Low Speed Access Module 15 To Core Routers Primary Rate T1/E1 PSTN lines to modem bank PSTN lines to built-in modems AS5400 2811 2800/3800 TACACS+/Radius proxy, DNS resolver, Content Web Cache Access Network Gateway Routers
  • 16. Medium Speed Access Module 16 To Core Routers Channelised T1/E1 64K and nx64K circuits Mixture of channelised T1/E1, 56/64K and nx64K circuits 3800/7206/7600
  • 17. High Speed Access Module 17 To Core Routers Metro Ethernet Channelised T3/E3 Channelised OC3/OC12 7200/7600/ASR1000/ASR9000
  • 18. Broad Band Access Module 18 To Core Routers Telephone Network The cable system 6400 SSG, DHCP, TACACS+ or Radius Servers/Proxies, DNS resolver, Content Web Cache Access Network Gateway Routers uBR7246 61xx IP, ATM
  • 19. ISP Services Module 19 DNS cache DNS secondary POP3 Mail Relay NEWS To core routers WWW cache Service Network Gateway Routers
  • 20. Hosted Services Module 20 Customer 7 Customer 3 Customer 4 Customer 5 Customer 6 To core routers Hosted Network Gateway Routers Customer 2 Customer 1
  • 21. Border Module 21 To core routers Network Border Routers To local IXP - NB - no default route + local AS routing table only ISP1 ISP2
  • 22. NOC Module 22 Primary DNS To core routers Hosted Network Gateway Routers SYSLOG server TACACS+ server Network Operations Centre Staff Out of Band Management Network 2811/32async NetFlow Analyser Firewall Billing, Database and Accounting Systems Corporate LAN Critical Services Module
  • 23. Out of Band Network 23 Out of Band Management Network 2811/32async To the NOC Out of Band Ethernet NetFlow Collector NetFlow enabled routers Router consoles
  • 25. Backbone Design  Routed Backbone  Switched Backbone  Virtually obsolete  Point-to-point circuits  nx64K, T1/E1, T3/E3, OC3, OC12, GigE, OC48, 10GigE, OC192, OC768  ATM/Frame Relay service from telco  T3, OC3, OC12,… delivery  Easily upgradeable bandwidth (CIR)  Almost vanished in availability now 25
  • 26. Distributed Network Design  PoP design “standardised”  operational scalability and simplicity  ISP essential services distributed around backbone  NOC and “backup” NOC  Redundant backbone links 26
  • 27. Distributed Network Design 27 POP One POP Two POP Three Customer connections Customer connections Customer connections External connections External connections Operations Centre Backup Operations Centre ISP Services ISP Services ISP Services
  • 28. Backbone Links  ATM/Frame Relay  Virtually disappeared due to overhead, extra equipment, and shared with other customers of the telco  MPLS has replaced ATM & FR as the telco favourite  Leased Line/Circuit  Most popular with backbone providers  IP over Optics and Metro Ethernet very common in many parts of the world 28
  • 29. Long Distance Backbone Links  Tend to cost more  Plan for the future (at least two years ahead) but stay in budget  Unplanned “emergency” upgrades can be disruptive without redundancy  Allow sufficient capacity on alternative paths for failure situations  Sufficient can be 20% to 50%  Some businesses choose 0% – meaning they have no spare capacity at all!! 29
  • 30. Long Distance Links 30 POP One POP Two POP Three Long distance link Alternative/Backup Path
  • 31. Metropolitan Area Backbone Links  Tend to be cheaper  Circuit concentration  Choose from multiple suppliers  Think big  More redundancy  Less impact of upgrades  Less impact of failures 31
  • 32. Metropolitan Area Backbone Links 32 POP One POP Two POP Three Metropolitan Links Metropolitan Links Traditional Point to Point Links
  • 33. ISP Services DNS, Mail, News Design and Placement 33
  • 34. ISP Services  Most ISP services such as DNS, Mail, etc are easily deliverable on low budget hardware platforms  Single Rack Unit in height (1RU)  Dual processor is “default” now  RAM is very cheap (may as well use 2Gbytes or more)  Hard drives are very cheap (SCSI more reliable)  Unix like operating systems (FreeBSD, Debian, Ubuntu, CentOS) are very common  In addition to commercial operating systems such as Solaris, RedHat Enterprise Linux &c  Minimal overhead, minimal OS install, plus the service required 34
  • 35. ISP Services: DNS  Domain Name System  Provides name and address resolution  Servers need to be differentiated, properly located and specified  Primary nameserver  Secondary nameserver  Caching nameserver – resolver 35
  • 36. ISP Services: DNS  Primary nameserver  Holds ISP zone files  Forward zone (list of name to address mappings) for all ISP’s and any customer zones  Reverse zone (list of address to name mappings) for all ISP’s address space  Hardware & OS: easily satisfied by simple specification  Located in secure part of net, e.g. NOC LAN  Usually run as “hidden master” – secondary nameservers are the official listed nameservers 36
  • 37. ISP Services: DNS  Secondary nameserver  Holds copies of ISP zone files  At least two are required, more is better  Hardware & OS: easily satisfied by simple specification  Strongly recommended to be geographically separate from each other and the primary DNS  At different PoPs  On a different continent e.g. via services offered by ISC, PCH and others  At another ISP 37
  • 38. ISP Services: Secondary DNS Example 38 $ dig apnic.net ns ;; ANSWER SECTION: apnic.net. 10800 NS ns1.apnic.net. apnic.net. 10800 NS ns3.apnic.net. apnic.net. 10800 NS ns4.apnic.net. apnic.net. 10800 NS ns5.apnic.com. apnic.net. 10800 NS cumin.apnic.net. apnic.net. 10800 NS ns-sec.ripe.net. apnic.net. 10800 NS tinnie.arin.net. apnic.net. 10800 NS tinnie.apnic.net. ;; ADDITIONAL SECTION: ns1.apnic.net. 3600 A 202.12.29.25 ns3.apnic.net. 3600 A 202.12.28.131 ns4.apnic.net. 3600 A 202.12.31.140 ns5.apnic.com. 10800 A 203.119.43.200 cumin.apnic.net. 3600 A 202.12.29.59 tinnie.apnic.net. 3600 A 202.12.29.60 ns-sec.ripe.net. 113685 A 193.0.0.196 tinnie.arin.net. 10800 A 199.212.0.53 Tokyo Hong Kong Washington Brisbane Brisbane Amsterdam Washington
  • 39. ISP Services: Secondary DNS Example  apnic.net zone  Primary DNS in Brisbane (ns1.apnic.net)  Secondary DNS run all over the world by APNIC:  Brisbane  Hong Kong  Tokyo  Washington  Zone secondaried by  RIPE NCC in Amsterdam  ARIN in Washington  Geographical and service provider redundancy – this is the perfect example! 39
  • 40. ISP Services: DNS  Caching nameserver  This is the resolver – it is the DNS cache  Your customers use this as resolver, NEVER your primary or secondary DNS  Provides very fast lookups  Does NOT secondary any zones  One, or preferably two per PoP (redundancy)  Hardware & OS: easily satisfied by simple specification 40
  • 41. ISP Services: Caching Nameserver  DIAL users automatically given the IP addresses of DNS caches when they dial in 41 To Core Routers DIAL network Web Cache DNS Cache DNS Cache Radius proxy Switch redundancy Router redundancy DNS Cache redundancy
  • 42. ISP Services: Anycasting the Caching Nameserver  One trick of the trade  assign two unique IP addresses to be used for the two DNS resolver systems  use these two IP addresses in every PoP  route the two /32s across your backbone  even if the two resolver systems in the local PoP are down, the IGP will ensure that the next nearest resolvers will be reachable  Known as IP Anycast 42 Geek Alert
  • 43. ISP Services: DNS  Efficient and resilient design  Primary DNS – keep it secure  Secondary DNS – geographical and provider redundancy  Don’t ever put them on the same LAN, switched or otherwise  Don’t put them in the same PoP  Caching DNS – one or two per PoP  Reduces DNS traffic across backbone  More efficient, spreads the load 43
  • 44. ISP Services: DNS  Software  Make sure that the BIND distribution on the Unix system is up to date  The vendor’s distribution is rarely current  Pay attention to bug reports, security issues  Reboot the DNS cache on a regular (e.g. monthly) basis  Clears out the cache  Releases any lost RAM  Accepted good practice by system administrators 44
  • 45. ISP Services: DNS  Implementation  Put all your hosts, point-to-point links and loopbacks into the DNS  Under your ISP’s domain name  Use sensible/meaningful names  Put all your hosts, point-to-point links and loopbacks into the REVERSE DNS also  Don’t forget about in-addr.arpa and ip6.arpa – many ISPs do  Some systems demand forward/reverse DNS mapping before allowing access 45
  • 46. ISP Services: Mail  Must have at least two mail hosts (MX records) for all supported domains  Geographical separation helps  Dedicated POP3 server  Consumers/mobile users get mail from here  SMTP gateway dedicated to that function  Consumers/mobile users send mail via here  Mail relay open to CUSTOMERS only!  Don’t let outside world use your mail relay  Block port 25 outbound for all customers  Insist that outbound e-mail goes through SMTP relay  SMTP relay does virus (ClamAV) and spam (Spamassassin) filtering 46
  • 47. ISP Services: Mail Configuration 47 smtp.isp.net Customer mail relay Incoming mail from customer mail.isp.net ISP Mail Gateway Incoming mail from Internet pop3.isp.net Customer POP3/IMAP server Mail pulled by customer client Mail out to the Internet SpamAssassin ClamAV SpamAssassin ClamAV
  • 48. ISP Services: Mail Example  cisco.com mail (MX records)  primary MX are 6 systems in San Jose  Three backup MXes in RTP, Amsterdam and Sydney  backup MX only used if primary unavailable 48 $ dig cisco.com mx ;; ANSWER SECTION: cisco.com. 86400 MX 10 sj-inbound-a.cisco.com. cisco.com. 86400 MX 10 sj-inbound-b.cisco.com. cisco.com. 86400 MX 10 sj-inbound-c.cisco.com. cisco.com. 86400 MX 10 sj-inbound-d.cisco.com. cisco.com. 86400 MX 10 sj-inbound-e.cisco.com. cisco.com. 86400 MX 10 sj-inbound-f.cisco.com. cisco.com. 86400 MX 15 rtp-mx-01.cisco.com. cisco.com. 86400 MX 20 ams-inbound-a.cisco.com. cisco.com. 86400 MX 25 syd-inbound-a.cisco.com.
  • 49. ISP Services: Mail  Software  Make sure that the MAIL and POP3 distributions on the Unix system are up to date  The vendor distributions are rarely current  Pay attention to bug reports, security issues, unsolicited junk mail complaints 49 IMPORTANT: Do NOT allow non-customers to use your mail system as a relay
  • 50. ISP Services: News  News servers provide a Usenet news feed to customers  Distributed design required  Incoming newsfeed to one large server  Distributed to feed servers in each PoP  Feed servers provide news feed to customers  Outgoing news goes to another server  Separate reading news system  Separate posting news system 50
  • 51. ISP Services: News System Placement 51 POP One POP Two POP Three Customer connections Customer connections Customer connections External connections External connections News Collector News Feeder News Feeder News Feeder News Distributor
  • 52. ISP Services: News System Placement 52 POP One POP Two POP Three Customer connections Customer connections Customer connections External connections External connections News Collector News Feeder News Feeder News Feeder News Distributor
  • 53. ISP Services: News  Software  Make sure that the Internet News distribution on the Unix system is up to date  The vendor distributions are rarely current  Pay attention to bug reports, security issues, unsolicited junk posting complaints 53 IMPORTANT: Do NOT allow non-customers to use your news system for posting messages
  • 55. Where to get IP addresses and AS numbers  Your upstream ISP  Africa  AfriNIC – http://www.afrinic.net  Asia and the Pacific  APNIC – http://www.apnic.net  North America  ARIN – http://www.arin.net  Latin America and the Caribbean  LACNIC – http://www.lacnic.net  Europe and Middle East  RIPE NCC – http://www.ripe.net/info/ncc 55
  • 57. Getting IP address space  Take part of upstream ISP’s PA space or  Become a member of your Regional Internet Registry and get your own allocation  Require a plan for a year ahead  General policies are outlined in RFC2050, more specific details are on the individual RIR website  There is no more IPv4 address space at IANA  Most RIRs are now entering their “final /8” IPv4 delegation policies  Limited IPv4 available  IPv6 allocations are simple to get in most RIR regions 57
  • 58. What about RFC1918 addressing?  RFC1918 defines IP addresses reserved for private Internets  Not to be used on Internet backbones  http://www.ietf.org/rfc/rfc1918.txt  Commonly used within end-user networks  NAT used to translate from private internal to public external addressing  Allows the end-user network to migrate ISPs without a major internal renumbering exercise  Most ISPs filter RFC1918 addressing at their network edge  http://www.cymru.com/Documents/bogon- list.html 58
  • 59. What about RFC1918 addressing?  List of well known problems with this approach for an SP backbone:  Breaks Path MTU Discovery  Potential conflicts with usage of private addressing inside customer networks  Security through obscurity does not provide security  Troubleshooting outside the local network becomes very hard  Router interface addresses are only locally visible  Internet becomes invisible from the router  Troubleshooting of connectivity issues on an Internet scale becomes impossible  Traceroutes and pings provide no information  No distinction between “network invisible” and “network broken”  Increases operational complexity of the network infrastructure and routing configuration 59
  • 60. Private versus Globally Routable IP Addressing  Infrastructure Security: not improved by using private addressing  Still can be attacked from inside, or from customers, or by reflection techniques from the outside  Troubleshooting: made an order of magnitude harder  No Internet view from routers  Other ISPs cannot distinguish between down and broken  Performance: PMTUD breakage  Summary:  ALWAYS use globally routable IP addressing for ISP Infrastructure 60
  • 61. Addressing Plans – ISP Infrastructure  Address block for router loop-back interfaces  Address block for infrastructure  Per PoP or whole backbone  Summarise between sites if it makes sense  Allocate according to genuine requirements, not historic classful boundaries  Similar allocation policies should be used for IPv6 as well  ISPs just get a substantially larger block (relatively) so assignments within the backbone are easier to make 61
  • 62. Addressing Plans – Customer  Customers are assigned address space according to need  Should not be reserved or assigned on a per PoP basis  ISP iBGP carries customer nets  Aggregation not required and usually not desirable 62
  • 63. Addressing Plans – ISP Infrastructure  Phase One 63 223.10.0.0/21 Customer assignments Infrastructure Loopbacks /24 223.10.6.255 223.10.0.1 223.10.0.0/20 Original assignments New Assignments /24 /24 223.10.0.1 223.10.5.255 223.10.15.255 Phase Two
  • 64. Addressing Plans Planning  Registries will usually allocate the next block to be contiguous with the first allocation  Minimum allocation could be /21  Very likely that subsequent allocation will make this up to a /20  So plan accordingly 64
  • 65. Addressing Plans (contd)  Document infrastructure allocation  Eases operation, debugging and management  Document customer allocation  Contained in iBGP  Eases operation, debugging and management  Submit network object to RIR Database 65
  • 67. Routing Protocols  IGP – Interior Gateway Protocol  carries infrastructure addresses, point-to-point links  examples are OSPF, ISIS,...  EGP – Exterior Gateway Protocol  carries customer prefixes and Internet routes  current EGP is BGP version 4  No connection between IGP and EGP 67
  • 68. Why Do We Need an IGP?  ISP backbone scaling  Hierarchy  Modular infrastructure construction  Limiting scope of failure  Healing of infrastructure faults using dynamic routing with fast convergence 68
  • 69. Why Do We Need an EGP?  Scaling to large network  Hierarchy  Limit scope of failure  Policy  Control reachability to prefixes  Merge separate organizations  Connect multiple IGPs 69
  • 70. Interior versus Exterior Routing Protocols  Interior  Automatic neighbour discovery  Generally trust your IGP routers  Prefixes go to all IGP routers  Binds routers in one AS together  Exterior  Specifically configured peers  Connecting with outside networks  Set administrative boundaries  Binds AS’s together 70
  • 71. Interior versus Exterior Routing Protocols  Interior  Carries ISP infrastructure addresses only  ISPs aim to keep the IGP small for efficiency and scalability  Exterior  Carries customer prefixes  Carries Internet prefixes  EGPs are independent of ISP network topology 71
  • 72. Hierarchy of Routing Protocols 72 BGP4 BGP4 and OSPF/ISIS Other ISPs Customers IXP Static/BGP4 BGP4
  • 73. Routing Protocols: Choosing an IGP  Review the “OSPF vs ISIS” presentation:  OSPF and ISIS have very similar properties  ISP usually chooses between OSPF and ISIS  Choose which is appropriate for your operators’ experience  In most vendor releases, both OSPF and ISIS have sufficient “nerd knobs” to tweak the IGP’s behaviour  OSPF runs on IP  ISIS runs on infrastructure, alongside IP 73
  • 74. Routing Protocols: IGP Recommendations  Keep the IGP routing table as small as possible  If you can count the routers and the point to point links in the backbone, that total is the number of IGP entries you should see  IGP details:  Should only have router loopbacks, backbone WAN point-to-point link addresses, and network addresses of any LANs having an IGP running on them  Strongly recommended to use inter-router authentication  Use inter-area summarisation if possible 74
  • 75. Routing Protocols: More IGP recommendations  To fine tune IGP table size more, consider:  Using “ip unnumbered” on customer point-to- point links – saves carrying that /30 in IGP  (If customer point-to-point /30 is required for monitoring purposes, then put this in iBGP)  Use contiguous addresses for backbone WAN links in each area – then summarise into backbone area  Don’t summarise router loopback addresses – as iBGP needs those (for next-hop)  Use iBGP for carrying anything which does not contribute to the IGP Routing process 75
  • 76. Routing Protocols: iBGP Recommendations  iBGP should carry everything which doesn’t contribute to the IGP routing process  Internet routing table  Customer assigned addresses  Customer point-to-point links  Dial network pools, passive LANs, etc 76
  • 77. Routing Protocols: More iBGP Recommendations  Scalable iBGP features:  Use neighbour authentication  Use peer-groups to speed update process and for configuration efficiency  Use communities for ease of filtering  Use route-reflector hierarchy  Route reflector pair per PoP (overlaid clusters) 77
  • 79. Security  ISP Infrastructure security  ISP Network security  Security is not optional!  ISPs need to:  Protect themselves  Help protect their customers from the Internet  Protect the Internet from their customers  The following slides are general recommendations  Do more research on security before deploying any network 79
  • 80. ISP Infrastructure Security  Router security  Usernames, passwords, vty filters, TACACS+  Disable telnet on vtys, only use SSH  vty filters should only allow NOC access, no external access  See IOS Essentials for the recommended practices for ISPs 80
  • 81. ISP Infrastructure Security  ISP server security  Usernames, passwords, TCP wrappers, IPTABLES  Protect all servers using routers with strong filters applied  Hosted services security  Protect network from hosted servers using routers with strong filters  Protect hosted servers from Internet using routers with strong filters 81
  • 82. ISP Infrastructure Security ISP Server Protection 82 DNS cache DNS secondary POP3 Mail Relay NEWS To core routers Service Network Gateway Routers Access-list examples: Allow tcp/established to all servers ICMP DNS 2ary: udp/53 and tcp/53 POP3: tcp/110 Mail Relay: tcp/25 and ISP address range only News: tcp/119 and ISP address range only DNS Cache: udp/53 Web server: tcp/80 Other necessary filters: All servers: SSH (tcp/22) from NOC LAN only Web server
  • 83. ISP Infrastructure Security Hosted Server Protection 83 Access-list examples: Inbound Allow tcp/established to all servers ICMP Web server: tcp/80 SSH for customer access Any other ports for services sold to customers Outbound ICMP Allow DNS udp/53 and tcp/53 Block all access to ISP address range Server5 Server1 Server2 Server3 Server4 To core routers Service Network Gateway Routers Server6
  • 84. ISP Infrastructure Security  Premises security  Locks – electronic/card key preferred  Secure access – 24x7 security arrangements  Environment control – good aircon  Staff responsibility  Password policy, strangers, temp staff  Employee exit procedures  RFC2196  (Site Security Handbook)  RFC3871  (Operational Security Requirements for Large ISP IP Network Infrastructure ) 84
  • 85. ISP Network Security  Denial of Service Attacks  eg: “smurfing”  see http://www.denialinfo.com  Effective filtering  Network borders – see Cisco ISP Essentials  Customer connections – unicast RPF on ALL of them  Network operation centre  ISP corporate network – behind firewall 85
  • 86. ISP Network Security Secure external access  How to provide staff access from outside  Set up ssh gateway (Unix system with ssh daemon and nothing else configured)  Provide ssh client on all staff laptops  ssh available on Unix and Windows  ssh is Secure Shell – encrypted link  How not to provide access from outside  telnet, rsh, rlogin – these are all insecure  Open host – insecure, can be compromised 86
  • 87. Ingress & Egress Route Filtering Your customers should not be sending any IP packets out to the Internet with a source address other then the address you have allocated to them! 87
  • 88. Out of Band Management 88
  • 89. Out of Band Management  Not optional!  Allows access to network equipment in times of failure  Ensures quality of service to customers  Minimises downtime  Minimises repair time  Eases diagnostics and debugging 89
  • 90. Out of Band Management  OoB Example – Access server:  modem attached to allow NOC dial in  console ports of all network equipment connected to serial ports  LAN and/or WAN link connects to network core, or via separate management link to NOC  Full remote control access under all circumstances 90
  • 91. Out of Band Network 91 Ethernet to the NOC Router, switch and ISP server consoles (Optional) Out of band WAN link to other PoPs Modem – access to PSTN for out of band dialin Equipment Rack Equipment Rack
  • 92. Out of Band Management  OoB Example – Statistics gathering:  Routers are NetFlow and syslog enabled  Management data is congestion/failure sensitive  Ensures management data integrity in case of failure  Full remote information under all circumstances 92
  • 94. Test Laboratory  Designed to look like a typical PoP  Operated like a typical PoP  Used to trial new services or new software under realistic conditions  Allows discovery and fixing of potential problems before they are introduced to the network 94
  • 95. Test Laboratory  Some ISPs dedicate equipment to the lab  Other ISPs “purchase ahead” so that today’s lab equipment becomes tomorrow’s PoP equipment  Other ISPs use lab equipment for “hot spares” in the event of hardware failure 95
  • 96. Test Laboratory  Can’t afford a test lab?  Set aside one spare router and server to trial new services  Never ever try out new hardware, software or services on the live network  Every major ISP in the US and Europe has a test lab  It’s a serious consideration 96
  • 98. Operational Considerations 98 Why design the world’s best network when you have not thought about what operational good practices should be implemented?
  • 99. Operational Considerations Maintenance  Never work on the live network, no matter how trivial the modification may seem  Establish maintenance periods which your customers are aware of  e.g. Tuesday 4-7am, Thursday 4-7am  Never do maintenance on a Friday  Unless you want to work all weekend cleaning up  Never do maintenance on a Monday  Unless you want to work all weekend preparing 99
  • 100. Operational Considerations Support  Differentiate between customer support and the Network Operations Centre  Customer support fixes customer problems  NOC deals with and fixes backbone and Internet related problems  Network Engineering team is last resort  They design the next generation network, improve the routing design, implement new services, etc  They do not and should not be doing support! 100
  • 101. Operational Considerations NOC Communications  NOC should know contact details for equivalent NOCs in upstream providers and peers  Or consider joining the INOC-DBA system  Voice over IP phone system using SIP  Runs over the Internet  www.pch.net/inoc-dba for more information 101
  • 103. ISP Design Summary  KEEP IT SIMPLE & STUPID ! (KISS)  Simple is elegant is scalable  Use Redundancy, Security, and Technology to make life easier for yourself  Above all, ensure quality of service for your customers 103
  • 104. ISP Network Design ISP Training Workshops 104