Skip to content

OpenDNS and speed

How to upset Xero’s Ops Team

In the Operations Team, we like to keep an eye on the Customer Care ticket queues, just to get an overview of any current problems. One of the worst things we can see is the very occasional complaint about Xero running slowly.

We’ve spoken in the past about our focus on delivering the fastest possible user experience in Xero, both in terms of application design and in our use of the Akamai Content Delivery Network. (CDN).

It’s fair to say we spend a lot of time and money on speed, and it’s actually hurtful to hear someone is having a bad experience with Xero.

We’re starting to see that the most common cause of speed issues with Xero is the rising popularity of Commercial DNS services like OpenDNS or Google’s Public DNS.

What CDNs do for Xero

To understand how these can cause problems, we need to look at how CDNs work, especially Akamai.

Akamai have over 55000 servers deployed around the world. To this extent, they’re unique in the CDN space in terms of scale. Their servers aren’t in large datacentres in major countries, they’re installed in the last few hops – right inside the ISP (Akamai call these servers the Edge).

When Xero is delivered to you, you don’t talk to Xero directly, but to these remote Akamai servers. Over 90% of the Xero app rarely changes – the images, the CSS, the JavaScript. That means if we store all this unchanging data in the Edge, we can deliver all of these really, really close to you, and your experience will be much faster.

The other really clever bit is how Akamai handle the data that does change. On the Internet, data moves over the most ‘economic route’, i.e the cheapest.

It’s a bit like driving from Paris to Marseille. You can take the N roads which are not tolled, but it’s going to be a lot slower. Using Akamai between the edge and our Rackspace environments is the equivalent of being able to drive on the A6/A7. We’re taking the fastest route available, not the cheapest deal your ISP has managed to negotiate. Akamai can even handle the occasional traffic jam – they check every 2 minutes the fastest route over the internet between the Edge and our environment, and makes changes as needed.

How the new DNS services break things

I’m not going to discuss how DNS works, but Wikipedia have a good article on how DNS works on your browser and computer.

To deliver the best experience, Akamai needs to select the very closest Edge server for your location. The trick to doing this, is to knowing where you are, before you first connect to an Edge Web-server. If you connect to an Edge server on the other side of the world, your experience is going to be much slower than one that is 2 hops away, directly on your ISP’s network.

The way to do this is to determine your location based on the DNS request you’ve made, which means the Edge server selected is based on what DNS server you use. If you use a DNS server from your local ISP, then you’re going to get a result that is close to those DNS servers. If you use OpenDNS, you’re going to get a result that is close to those servers – not one that is necessarily close to you.

It’s not just Xero that is going to be impacted by this, but many large sites that you use every day, and the problems are not just that some sites will be slower – but sometimes, they won’t work at all.

DNS services such as OpenDNS and Google’s Public DNS have some advantages.  Unfortunately, using them won’t just affect your Xero experience, but also the huge number of sites using Akamai and other CDN networks.

 

Read more about Company News

 

12 comments

elliot
10 December 2010 #

Thanks for this post guys, cleared up not only why xero is sometimes slow, but why many USA hosted sites load one day and not the next. Surely though no matter how mega fast your host is, if its in USA a NZer is going to have a slow experience because of nz’s one tiny line out and one tiny line in for whole countries internet.

Paul Rushworth
10 December 2010 #

Hi Elliot,

A NZ Xero user, with properly configured ISP DNS will be talking to NZ based servers- that’s the whole point of Akamai. There is decent connectivity into NZ, but it is very expensive- which is why the connection Akamai Edge servers provide to our Rackspace environments is so crucial.

Hopefully the guys at Pacific Fibre will be doing their best to address this problem in the near future, to improve performance for all sites….

Jakub S
10 December 2010 #

Two insights: There are CDN networks using routing mechanisms instead of old school DNS out there. I certainly feel that’s a better way.

The only reason for the popularity of Google’s DNS (I use it myself) or OpenDNS is the slow speed of ISP DNS services. In my case of BT in UK, the DNS is in almost 100 % of the cases a root of a problem why your internet “doesn’t work”. Switch to 8.8.8.8 and you are fine…

Paul Rushworth
10 December 2010 #

Hi Jakub,

Depending on how your local network is setup, (perhaps you’re using DHCP off your modem/wireless router – If your DNS server is set to a private address, you’re probably using the DNS resolver on your Wireless Access Point/xDSL Modem. )

There are many routers out there of varying quality- It’s really not uncommon for the problem to be a poor quality dns resolver on a under spec DSL Router.

If changing the clients computers DNS to OpenDNS or Google fixes this problem, then its worth look at the Router as a potential cause, it’s not always the ISP.

Wayne Robinson
10 December 2010 #

Of course, in many popular locations this isn’t really a problem because DNS services like OpenDNS and Google’s DNS product have servers that are vet close to the end user, often closer than their own ISP and chosen via AnyCast.

However, because we are in the arse end of the world within NZ and Australia which none of these providers consider to be of any value, we get stuck with the closest servers being in California or Singapore, not significant trips across the ocean.

However, I thought Xero just used their CDN to serve up static content which should be long-term cached in the browser anyway and that your app servers were located in a Rackspace datacentre somewhere in the USA, which is where all dynamic data requests will go and have a minimum 250-ish millisecond request round trip anyway.

Paul Rushworth
10 December 2010 #

Hi Wayne,

Xero has used Akamai’s Web Application Accelerator for a little over 12 months now- it is a lot more than just static content pushed out via CDN.
Page 4 of http://www.akamai.com/dl/brochures/akamai_waa_sb.pdf has some details. Point 3 is the really interesting bit, can’t elaborate on the specifics, but I left the first technical meeting that Craig and I had with Akamai thinking ‘this is the coolest tech EVER’.

Unfortunately, not all browsers will cache SSL served content, despite any cache headers being set by the server, so there is still a benefit in this for some users.

Wayne Robinson
10 December 2010 #

Don’t get me wrong, Akamai is really cool and you can do some really interesting dynamic-based serving and even dynamic caching with it however, I would’ve thought the majority of your dynamic pages were server-generated because most of them would have to hit your database server (e.g. invoice list, invoice detail, etc).

Paul Rushworth
10 December 2010 #

Correct- but in terms of data volume, this is only ~10% of all site traffic.

Adam Clarke
10 December 2010 #

Also worth noting that some ISPs (eg Xnet) treat Akamai contest as local, and thus it won’t count against usage caps.

Martin Barry
10 December 2010 #

There has been some proposals to try to fix this but they have not made much progress. The “DNS tricks” are seen as breaching the intent if not the letter of the protocol and any additions to provide assistance in that regard are frowned upon.

Having said that there is nothing stopping OpenDNS or Google working with CDN vendors to work around this in a non-standard way but clearly that does not scale and creates barriers to entry for new CDNs and third party DNS providers.

iphone app development co
10 December 2010 #

I suspect Xero might not be the only thing running slowly at the moment as the DDOS war between Wikileaks supporters and detractors sucks up lots of bandwidth.

Steve Del-Nevo
11 December 2010 #

Hi Guys

We have been using OpenDNS for a long time now and we have not noticed any down turn in speed when using it (perhaps this is because our ISP DNS servers are useless) but the main reason for us using OpenDNS is the fantastic level of content filtering that this product provides.

Perhaps people are over looking the productive benefits that small business get from filtering websites such as face book,I personally would rather Xero rang very slightly slower (if it indeed does) than have my staff updating there status every 5 minutes instead of supporting our clients.

Add your comment





We welcome all feedback but prefer a real name and email address.