How is the internet really wired?

In order to better understand the workings and potential of the Internet,Ingenia has commissioned a series of articles that will outline some of the engineering behind the global network. John Souter of the London Internet Exchange starts by describing the physical infrastructure that transports web traffic.

Switching: LINX uses two types of switch from different equipment vendors; these are mounted in racks with line cards fitted and wired ready for service

The Internet is not a single entity. It is a network of networks, bound together by the goal of reaching everyone, everywhere. We take it for granted that we can surf the Internet doing what we want, when we want, regardless of our location.

We email and read content of unprecedented scope and scale. We chat to friends worldwide via instant messaging, trade, game and socially network with just about anyone we choose. So how does it all work, how do your email messages get from your Internet service provider (ISP) to the ISP of the recipient? How is the Internet wired?

Access and Content

Access networks form a crucial part of the Internet, aiming to provide end-users with access to just about anything, anywhere online. These networks typically consist of real wires or fibres in the ground, and more recently wireless systems, together with transmission equipment to move Internet traffic from one point to another.

Meanwhile, content networks, a second type of network within the Internet, consist of stores of information or interactive content that users wish to access. Google, for example, has a content network that comprises e-mail, digital location maps, office productivity, social networking, video sharing services and many other things, as well as being a search engine.

Clearly content network operators want their network to be reachable by everyone, which means a robust connection between content and access networks is critical. To achieve this, network operators have two choices.

Buying or peering?

First, the operator can buy connectivity from an ‘upstream provider’, who usually manages a bigger and more widely distributed network. In such a ‘transit’ arrangement, the upstream provider takes care of connecting traffic to the rest of the Internet.

A second option, known as ‘peering’, allows network operators to establish relationships with other operators so that they can exchange traffic destined for each other’s networks. This suits many ISPs and network operators who wish to have more control over their destinies, rather than simply entrust their precious data to upstream transit providers. This set-up also appeals to many, as peering is typically carried out ‘settlement free’ with no money changing hands in either direction.

Two separate forms of peering exist. ‘Private peering’ takes place on a one-to-one basis between two network operators and is the direct physical connection to another network, usually in data centres where the relevant equipment of the two parties is housed. ‘Public peering’ involves connecting one network to many networks and takes place at an Internet exchange.

Exchanging traffic

The physical infrastructure that allows network operators to exchange traffic directly between each other is called an Internet exchange point. Direct interconnection brings numerous benefits but the key advantages are reductions in cost and latency (the time taken for traffic to travel between the two peering networks).

Internet exchanges typically deploy one or more large Ethernet network switches. Ethernet is a family of frame-based computer networking technologies for local area networks (LANs). The Ethernet switches can read the ‘header’ – analogous to address labels – of the incoming data packets and forward as required. These switches can be as big as a wardrobe and consist of many ports to which the peering ISPs and network operators can connect their networks. The capacity of each port is crucial, with today’s largest switches containing up to 128 ports each capable of transferring data at 10 gigabits a second (Gb/s).

Once connected to a port, the ISP or network provider will set up routers to automatically exchange traffic according to the Border Gateway Protocol – the Internet’s core routing protocol. While the Internet exchange will monitor these arrangements, ensuring traffic from all ports can potentially reach and then be transferred to all other ports on its switch-based infrastructure, it will not typically have a direct role in the actual peering. However, one important exception exists: ‘multi-lateral peering’.

Multi-lateral peering – whereby every network operator using this service can peer with every other network operator – can be set up by using a route server. This is a server running dedicated routing software to which network operators must also connect. While such a setup requires all involved to have very open peering policies, the automation it offers can reduce the social or human costs of peering and is becoming more and more important as the number of participants at Internet exchanges increases.

The London Internet Exchange

The London Internet Exchange (LINX) is one of the world’s largest and most successful Internet exchanges. At the start of 2009, peak traffic across its networks exceeded 500 Gb/s, with total connected capacity surpassing two terabits per second (in other words, at 25% of the capacity being used).

This not-for-profit organisation operates as a ‘mutual’ exchange, and is currently owned by more than 300 ISPs, telecommunications businesses, content and content delivery networks from around the world. Indeed, networks from nearly 50 different countries are represented at LINX.

The exchange itself has 10 so-called ‘points of presence’ or access points located among the data centres of the London area, from Docklands to as far as Slough to the west of London. It consists of two very separate public peering networks, based on switches from two different equipment vendors.

At least two switches, one from each vendor, are installed at each point of presence, with these locations connected by several 10 Gb Ethernet circuits to form two separate networks. As such, the networks operate independently and provide back-up for the other if necessary, as both systems should not develop the same fault at the same time. Indeed, most LINX members connect to both switching platforms, which reduces the impact of any downtime on a single network element.

Network Topology

The London Internet Exchange (LINX) Network consists of two separate high-performance Ethernet switching platforms installed across 10 locations across London. LINX does not own any of these sites; at each they are tenants in some form of co-location facility or carrier hotel. LINX is one of the UK’s major users of such facilities and are probably one of the most technically advanced users.

Switches from different equipment vendors, Extreme Networks and Brocade, are deployed in two diverse networks to provide an extra level of fault-tolerance, the logic being that both systems should not develop the same fault at the same time.

Therefore, at least two switches (one from each vendor) are installed in every LINX location and the 10 locations are interconnected by multiple 10 gigabit Ethernet circuits (across dark fibre) to form two physically separate backbone rings (one with Brocade equipment, the other Extreme).

Establishing LINX

So how was LINX set up? The development of the exchange can be traced back to when two ISPs, PIPEX and UKnet, linked their networks via a 64kb serial link to avoid the astronomical bandwidth cost and time-delay involved in routing data across the Atlantic to US Internet exchanges.

When Demon Internet, UKERNA (the UK academic network) and other ISPs showed interest in establishing similar serial links, Keith Mitchell, then Chief Technical Officer of PIPEX, initiated a meeting with BT to discuss the creation of a London-based Internet exchange. BT agreed to join the co-operative.

And so in November 1994, using a donated piece of equipment no bigger than a video recorder and without any legal contracts, the five UK-based ISPs linked their networks. This moment marked the birth of the London Internet Exchange.

PIPEX first provided the LINX founders with a Cisco Catalyst 1200 switch with eight 10 Mb ports, and rack space was leased at a virtually empty data centre in London’s Docklands. However, in the summer of 1996, LINX became the world’s first exchange to deploy a 100 Mb switch, a Cisco Catalyst 5000.

In January 1999 the exchange set up a Metropolitan Area Network – a city-sized computer network – running over gigabit Ethernet connections. And in 2002, LINX became the first exchange to introduce 10 Gb Ethernet operation, the latest in network technology.

Secrets of success

LINX attributes its success to several factors, including the economics of peering. When peering works well it can save money. The cost of connection, for a given size of connection port, is essentially fixed, so the more traffic one can peer over a port, the more money can be saved. While transit arrangements are now a lot cheaper than in the past, the fixed cost of peering remains attractive to ISPs and network operators.

In a transit arrangement, it can be catastrophic should something go wrong with an upstream provider or if connectivity is lost to the wider Internet – especially in terms of customer confidence. On the other hand, losing one or more peering connections doesn’t have anything like the impact, and may be completely invisible to most customers.

London itself is an advantageous place to site an Internet exchange. It houses an incredible number of networks to connect to and has high capacity connections to rival just about anywhere in the world.

Operating a mutual exchange is important, too. Many competitors cooperate at Internet exchanges, so having confidence in an exchange’s technical infrastructure, as well as its economic and policy base, is key. What’s more, LINX’s high concentration of peering participants also attracts additional peers and helps to keep costs down.

What lies ahead?

So what is the future for LINX, and indeed Internet exchanges worldwide? A key challenge is to keep the spirit of a mutual exchange healthy. As exchanges worldwide expand into new geographic locations, taking on more and more ISPs and network operators, they will by nature become a little less intimate.

Local meetings and video conferencing may prove crucial to ensuring exchanges remain in contact with their peering network participants. And in light of these increasing numbers, multi-lateral peering will play an increasingly important role in the future.

Having pioneered the use of the 100 Mb, 1 Gb and 10 Gb Ethernet standards, the next move for LINX is to trial the 100 Gb standard, which promises faster Internet speeds than ever before. This iteration of the Ethernet standard is not expected to be fully ratified and published until 2011 at the earliest, but equipment vendors are already working on the next generation of switches which should be ready for testing next year.

Testing, testing

Early trials of tomorrow’s switches are crucial. Looking at LINX itself, the exchange already comprises a complex network of switches, with 13 switches on a first Local Area Network and 12 on the second.

To avoid traffic bottlenecks at the exchange, these switches must be connected together with adequately sized ‘inter-switch links’. The exchange’s widest inter-switch link uses a technology called link aggregation to group together eight 10 Gb ports into a single virtual connection. Using several ports in this way increases the link speed beyond the limits of any single port, but in doing so the exchange is edging closer to the maximum capacity that the current generation of switch can provide.

LINX members currently use more than 600 ports, with over 150 of these being 10 Gb in capacity. Add that value to the large number of ports LINX uses for inter-switch links and it becomes clear that switch port capacity is a huge issue, and higher density ports will be needed soon.

Internet exchanges play a key role in making the Internet work, and in assisting both network operators and content providers to connect together efficiently. The number of Internet exchanges is growing, as is both the traffic that they handle and the number of networks they serve. The challenge of constantly growing traffic and new uses being found for the Internet that cause traffic growth are ever-present but the new Ethernet standard should help to address this.

Further reference
http://www.linx.net/
http://www.euro-ix.net/

The author would like to thank Dr Rebecca Pool for her help with this article

BIOGRAPHY – John Souter

Mathematics graduate John Souter joined LINX as chief executive in March 2001. John had a 20 year career with the British Standards Institution, culminating in his directorship of BSI Quality Assurance. After spells at Mobile Systems International plc and AAH Meditel Ltd, he was appointed the UK Managing Director of Varetis Communications in 2000, which provides Internet, WAP-based and computer systems to telephone companies. He is co-author of a technical book on Portable Modula-2 Programming and is a member of the editorial board of Software Quality Journal.