Lessons from IPv6 Day

Editor's Notes

• #2: This talk covers some of the things we leant as a result of participating in Wold IPv6 Day on 8th June 2011. It’s presented mainly from a server administrator’s point of view and, while it mentions assorted network-level issues, it doesn’t go into particular detail. In particular it’s not a guide to setting up an IPv6-capable network, nor a primer on what IPv6 is.
• #3: We are probably all used to IPv4. Been around for ages. Critically uses 32 bits to represent addresses, normally written as four dot-separated octets, each expressed in decimal. Trouble is, the world is running out of IPv4 addresses (all the ‘spare’ has now been allocated for use, though there are still addresses not actually being used). IPv4 is only surviving thanks to extensive use of RFC 1918 ‘private’ addresses, though their properties mean that ever increasingly complicated workarounds are needed to support their continued use.
• #4: IPv6, on the other hand, uses 128 bits to represent addresses (and note that doesn’t mean that the address space is only four times bigger...), normally written in hexadecimal as multiple 16-bit blocks sepearted by ‘:’ and with rules allowing runs of zeroes to be omitted. The two protocols have quite a few other differences, some of which we’ll come on to, but the longer addresses are the ones you see first. This is an example address used on IPv6 day...
• #5: ...except that the University has recently been allocated a new, bigger address range (/44 prefix in place of a /48) which means that all the addresses have to change.
• #6: So, what was IPv6 day all about?
• #7: Here’s what the Internet Society (who suggested and promoted the idea) have to say on the subject.
• #8: Here are some of the big players who started it off by promising to take part. Most of these already made their services available over IPv6, though not by default. In the end, at least 1,000 other providers, including the University of Cambridge, also joined in.
• #9: We gave this some thought in advance, and identified a number of things that we’d need to worry about...
• #10: IPv4 (at least in Cambridge where DHCP - especially dynamic DHCP - has always been considered a bit iffy) needs manual configuration: address, netmask, router, etc. v6, on the other hand, will by default try to configure itself. Connect any modern OS to many IPv6-capable networks and the machine will acquire a globally-routable address. This difference can lead to some surprises.
• #11: The DNS handles name<->IPv4 mapping separately to name<->IPv6 mapping So there’s no guarantee that you’ll hit the same server, never mind the same service, over v6 as over v4. Setting things up like this may lead to madness, but can sometimes be useful. IPv6 config may be needed at an application level - for example Apache needs to know what IP addresses it’s doing name-based virtual hosting on and so will need to know about v6 addresses as well as v4 ones. If an advertised v6 address isn’t responding (perhaps because the v6 interface is down) but the corresponding v4 interface is responding then clients will tend to try v6 and only fall back to v4 after a timeout. The symptoms can look VERY like server or network overload!
• #12: Packet filters and firewalls will need new configuration for v6 - default will probably be to block everything or allow everything, neither of which will probably be what you want.
• #13: It’s tempting to consider a machine with a RFC 1918 private address behind a NAT service to be more secure that a publically addressed one, because it can’t be poked directly from the outside. Private v6 addresses do exist, but are not widely deployed because they are typically a solution to an address shortage and we are not short of v6 addresses. So, stick a v4 privately-addressed machine on a subnet that also supports v6 and it will probably be out there exposed on the public Internet with a global address. This may come as a surprise.
• #14: It’s common to setup inter-host communications (e.g. web server to database) to use the localhost interface and to limit connections to this to prevent external meddling. But if you enable v6 on such a machine then internal connection may happen via the v6 local interface on ::1 and not the v4 one on 127.0.0.1. If your rules don’t take this into account you may find that you can’t talk to yourself.
• #15: Rather a lot of log analysis software may be assuming that IP addresses in logs will look like 131.111.10.33, and may be ‘surprised’ to find ones that look like 2001:630:212:8080::80:0. How they react will vary, but ignoring such entries (perhaps silently), or stoping dead on the first one are both possibilities.
• #16: ...and once we got into actually doing the necessary configuration we found some others:
• #17: If an IPv6 router finds it has a packet that’s too big to send over a particular link it drops the packet and sends a ‘Packet too big’ ICMP6 message to the packets origin, which is expected to resend it smaller. If anyone foolishly blocks those ICMP6 messages then this won’t work, and you’ll find that you can successfully send small packets but not full size ones. In a web context, this can mean that clients can open connections and successfully send requests, but can’t receive responses (which are typically much bigger). IPv6 requires that all links carry at least 1280 byte packets (c.f. 1500 byte packets typically used on Ethernet) and there is some evidence that the big providers are artificially limiting themselves to 1280 bytes, presumably to avoid this problem. [IPv4 also has fragmentation, but it handled on a per-link basis, rather than end-to-end. It too can cause problems, but these are now largely understood and normally avoided]
• #18: Even though it’s been around for a while, IPv6 is still changing quite rapidly, and even ‘current’ software may not be keeping up. For example all but the most recent point release of the version of MacOS current on IPv6 Day had a bug that was likely to affect some users. SuSE Linux Enterprise 10 (old, but still in support) has some failings in its v6 support that caused us problems.
• #19: The core of the CUDN already supports IPv6, as does JANET, but only a few University edge networks have enabled it (UCS, Astronomy, Computer Lab, SRCF, ...). The plan was to enable IPv6 on all these services for Pv6 day...
• #20: ...but inevitably some fell by the wayside. We did manage the rest.
• #21: No known problems experienced by any University clients accessing v6-enabled services.
• #22: A small but significant number of people accessed our v6 enabled services, apparently successfully.
• #23: OK, not exactly big numbers. Services mainly offered to internal clients likely to be low because of the small number of internal clients with IPv6 connectivity. For services also accessed from outside ( www.cam , mx.cam) ~1% of accesses were over v6.
• #24: China/Brazil probably high because the developing world has disproportionately fewer IPv4 addresses then US/Europe, etc., because by the time they wanted them the shortage was already becoming apparent and allocation rules were tightened. Such countries are likely to already be deploying v6 to cope with this.
• #25: Because of the disconnect between IPv4 and IPv6, various people have created systems what will, automatically or with manual configuration, allow v4 and v6 hosts to communicate or allow a pair of v6 hosts that don’t have v6 connectivity between them to communicate. ‘6to4’ is one such, and a common bug is that machines will sometimes chose an IPv6 connection via one of these ‘transitional technologies’ in favor of a ‘real’ IPv4 connection. For example lots of clients in the University contacted www.cam and smtp.hermes over 6to4 even though all those clients will have had viable IPv4 routes to the same servers. This causes some problems.
• #26: 6to4 is really clever, and here’s a diagram of how it works. You might want to look at the Wikipedia description for more detail: http://en.wikipedia.org/wiki/6to4 The critical points are that a 6to4 host ends up with an entirely usable IPv6 address in the 6to4 range 2002::/16, and if it wishes can offer to route other address in that range on behalf of other clients on the same subnet (thus bringing IPv6 support to a network that wouldn’t otherwise have it). But all this depends on connections that are probably crossing the institution boundary and which are probably being offered on a ‘best efforts’ basis at best.
• #27: So now you have machines on your network that are using IPv6 addresses from a range that you don’t expect. Any access control by IP address is likely to be messed up by this. Worse, since 6to4 machines can advertise themselves as IPv6 routers to other machines, the existence of a machine doing this can easily affect other machines on the same subnet. We saw this effect on IPv6 Day. Part way through the day a department mail server suddenly started using a 6to4 connection being offered by a workstation on the same network. Unfortunately it was forwarding mail to the central mail switch which refused to accept it because it wasn’t (apparently) coming from a machine in the University. Fortunately this was easily fixed, and didn’t result in a loss of mail, but does suggest that a significant barrier to wider Pv6 deployment may turn out to be these very ‘transitional’ technologies that were designed to make it easier.
• #28: The bottom line from IPv6 day is that enabling ‘dual stack’ (IPv6 alongside IPv4) operation on servers ‘just works’ and generally doesn’t cause problems for clients (which may themselves be v4-only, v6-only, or dual stack). However 6to4 (and similar technologies), when used inappropriately, may cause problems for some IP address-based access control systems. By and large adding IPv6 support to new or existing servers on networks that already support IPv6 is not difficult.