How to upset Xero’s Ops Team
In the Operations Team, we like to keep an eye on the Customer Care ticket queues, just to get an overview of any current problems. One of the worst things we can see is the very occasional complaint about Xero running slowly.
It’s fair to say we spend a lot of time and money on speed, and it’s actually hurtful to hear someone is having a bad experience with Xero.
What CDNs do for Xero
To understand how these can cause problems, we need to look at how CDNs work, especially Akamai.
Akamai have over 55000 servers deployed around the world. To this extent, they’re unique in the CDN space in terms of scale. Their servers aren’t in large datacentres in major countries, they’re installed in the last few hops – right inside the ISP (Akamai call these servers the Edge).
The other really clever bit is how Akamai handle the data that does change. On the Internet, data moves over the most ‘economic route’, i.e the cheapest.
It’s a bit like driving from Paris to Marseille. You can take the N roads which are not tolled, but it’s going to be a lot slower. Using Akamai between the edge and our Rackspace environments is the equivalent of being able to drive on the A6/A7. We’re taking the fastest route available, not the cheapest deal your ISP has managed to negotiate. Akamai can even handle the occasional traffic jam – they check every 2 minutes the fastest route over the internet between the Edge and our environment, and makes changes as needed.
How the new DNS services break things
I’m not going to discuss how DNS works, but Wikipedia have a good article on how DNS works on your browser and computer.
To deliver the best experience, Akamai needs to select the very closest Edge server for your location. The trick to doing this, is to knowing where you are, before you first connect to an Edge Web-server. If you connect to an Edge server on the other side of the world, your experience is going to be much slower than one that is 2 hops away, directly on your ISP’s network.
The way to do this is to determine your location based on the DNS request you’ve made, which means the Edge server selected is based on what DNS server you use. If you use a DNS server from your local ISP, then you’re going to get a result that is close to those DNS servers. If you use OpenDNS, you’re going to get a result that is close to those servers – not one that is necessarily close to you.
It’s not just Xero that is going to be impacted by this, but many large sites that you use every day, and the problems are not just that some sites will be slower – but sometimes, they won’t work at all.
DNS services such as OpenDNS and Google’s Public DNS have some advantages. Unfortunately, using them won’t just affect your Xero experience, but also the huge number of sites using Akamai and other CDN networks.