Skip to content

Infrastructure upgrade

This morning we moved to version 3 of our hosting infrastructure.

We started planning for this move in January. This is the next natural step up in our hosting capability and puts us in an environment to continually scale and grow.

For the geeks out there, highlights of what we did last night included:

  • Mixture of diagonal and vertical scaling of hardware for increased capacity and performance
  • Moved our data store from attached storage to a RAID 10 SAN
  • Virtualized some devices (our own dedicated virtualization environment) to allow us to better scale out our service delivery platform (the Xero application is still running on completely dedicated hardware)
  • Moved to a completely 64 bit platform running on Windows 2008
  • Upgraded our database management system to SQL Server 2008

Projects of this magnitude require a huge amount of planning and testing so thanks to the team for all the up front work that enabled this major platform upgrade to proceed without a hitch.

 

Read more about Company News

 

21 comments

Bart
31 August 2009 #

As a Geek I can appreciate the amount of work that went into planning this upgrade, a good job well done – give them a raise I say!

Dermott Renner
31 August 2009 #

Good news but do you have any plans to host locally or regionally i.e. NZ or Australasia as there are times when we use Xero itself where it so slow that it times out; I am talking over 2 minutes here.

All the big players Microsoft, Google host applications regionally.

From the US Xero is very fast but from NZ at times it runs like a snail.

As I am in US at moment I cannot say whether the above changes have helped speed in NZ.

Wayne Robinson
31 August 2009 #

A 2 minute time-out would have nothing to do with any latency at Xero’s end. Hosting in the USA only adds about 200ms to the round-trip time and, from what I have read, they are using regional content distribution for static content anyway (images, JavaScript libraries, etc). For general usage, the latency should be almost invisible. A 2 minute timeout seems like a network problem at your end, rather than a problem with Xero’s infrastructure.

Paul.Rushworth
1 September 2009 #

Bart,
Thanks- there certainly was a lot of planning and work throughout the implementation of this project. I remember not being able to get back to sleep after the 2am Wellington Earthquake last week- double checking every detail in my mind…

Dermott,

Performance of the app is something we take very seriously. Craig has spoken about our approach in the past: http://blog.xero.com/2008/07/optimising-xero-performance/

Wayne,

You’re correct: Xero does use a regional content delivery system to improve performance in key markets. We’re also looking at a couple of new technologies that should provide even faster performance for our customers.

John Younger
1 September 2009 #

Hi Craig
I am interested in : “Mixture of diagonal and vertical scaling of hardware for increased capacity and performance”

Can you elaborate a little?

Thanks
JY

Craig Walker
1 September 2009 #

You probably already know this but vertical scaling is basically scaling up – we were running on some pretty small hardware (especially on our database server) so we definitely needed to add some more cores, RAM and a lot more disk (who knew accounting data would take up so much room?).

Diagonal scaling is a term I first heard used at Flickr – essentially instead of scaling up or out you scale by upgrading to better & newer hardware often reducing the need for a lot of servers but giving you a similar boost in scale & performance. In Flickr’s case they replaced 67 dual socket machines with 18 dual socket quad-core machines and recovered almost 4x rack space and reduced costs by about 50 percent.

In our case it was a matter of faster & newer processors with more cores & cache, faster RAM, a tighter control on configuration (especially for our database server), faster network (you might have gigabit on your server network adapters but what do your firewalls & load balancers have?) and upgrading to Windows 2008 & SQL Server 2008 (we were previously running on Windows 2003 & SQL Server 2005).

So we haven’t really added any more servers but have added a lot more capacity.

Now I think I need to diagonally scale my Mac Pro …

Craig

Dermott Renner
2 September 2009 #

Sorry for the delay overseas

@Wayne
The problem is not our line we have an ISP grade fibre line high speed etc

I think the issue I raised is misunderstood by some of the people who replied; all the processors in the world etc will not overcome the lack of regionally hosted the data. There would be many advantages commercially for Xero doing this as well as speed.

Stephen Carr
8 September 2009 #

@Dermott

I too from time to time have very slow access to Xero, generally the same time when Mobile Me takes an age to do anything. Australasian hosting of Xero would be fantastic, considering it is a Kiwi company.

Brynn Neilson
8 September 2009 #

“A 2 minute timeout seems like a network problem at your end, rather than a problem with Xero’s infrastructure.”

We used to host in the States but have since moved everything to NZ because of the following:

1. It is faster. New internet traffic (non-cached) such as DB queries and SSL traffic is normally 128 – 256Kbits through major NZ ISPs DSL so if your servers were here local customers would have 8Mbit traffic.

2. More Hops = more problems. If I do a traceroute to an NZ IP address I can expect around 10 hops. When I traceroute Xero it is 15 hops which gives the opportunity for more points of failure.

3. Local access IS better. Cloud infrastructure such as Amazon’s EC2 also appreciate these issues and clusters servers Internationally so if I access the site from the US it uses a US node; if I access the site from the UK it uses the UK node.

We’re presently structuring our servers to be EC2 compatible so when we do finally get an EC2 node in NZ we can expand our servers through their cloud.

Don’t know how this would all work with Windows though. We only use Open Source tech: http://brynnneilson.blogspot.com/2009/06/free-software-is-better-for-business.html

brynn :-)

jake
8 September 2009 #

Local hosting is extremely expensive given the high cost of bandwidth here – i think Xero is doing the correct thing by hosting with Rackspace. They are the best hosting company worldwide and i dont believe they have a rival on this side of the atlantic.

Liam Farr
8 September 2009 #

@Brynn

As a network operator I disagree. Local hosting is actually quite cheap, and national bandwidth very much so. (It’s the international CIR that’s pricy).

I see Xero as a premium product, and hence find it supprising that it’s not sitting on the national backbone. (I haven’t trace routed the servers to verify it’s intl as I’m sitting in a restraunt on my iPhone).

Liam. (nzl@me.com).

Caelen
8 September 2009 #

Local hosting may start becoming a necessity for EU customers due to data protection legislation. Fundamentally this means you can’t store data about someone outside of the EU without their permission. Given that invoices frequently hold personal information it would seem that have local hosting in the EU for data would be prudent.

Dermott Renner
9 September 2009 #

@jake
I am not asking for all of Xero to be hosted in NZ or even Australasia. All I think is US customers access US servers, NZ/Aus servers in this part of the world. This is not a religious argument here; it will be faster for NZ/Aus customers if it is closer; basic physics.

Local hosting is not expensive in my opinion

Craig Walker
9 September 2009 #

Some great comments in here – hopefully I can cover off most of the points that have been raised.

First off let me just say that even though there are some great hosting companies in New Zealand (we use Revera for our internal environment and they’re fantastic), they still don’t hold a candle to RackSpace which operates a global company with global scale. I’m not trying to put down any of the NZ hosters – I personally know a great deal of people that work in those companies and they’re all amazing at what they do – but at RackSpace we have a dedicated team providing 24/7 support, 100% guaranteed availability, multi-path everything (from hardware to power to network), 1-hour full hardware replacement, often sub-minute response times to issues & (literally) hundreds of certified engineers at our disposal for anything we deem important (which, of course, is everything!). Even though hosting in NZ can be quite cheap you cannot get the level of service provided by a company with the size of RackSpace for the same cost – the market just doesn’t support it.

Now obviously it’s been suggested that we still use RackSpace, but that we also geographically distribute dynamic data (i.e., the accounting data) as well as the static stuff we already do. That’s actually a lot more difficult than it sounds. At the moment we have the concept of a Xero user account that allows you to access many organizations using the one login. Where do we host the login? What if you have a organization in NZ, Australia and the UK (which a lot of our customers do)? What if you travel? Do you expect your data to move with you on that flight to the UK?

Obviously some of these concerns don’t affect all our customers – but there is enough complexity there with the customers that are affected that coming up with a solution that works for everyone is very difficult. In fact it’s so difficult only a few companies (and I’m talking companies with limitless resources like Google) have tried to tackle while still maintaining a seamless user experience.

Now that’s not to say that we might never do it – never say never. But with the rapidity of our release cycles and the features we’ve still got to deliver we’re going to need to make sure we can do it in a way that fits with our goals of building the world’s easiest accounting system.

So in the meantime we’re doing everything we can to make the experience as great as possible with the resources and infrastructure we’ve currently got. It’s a never-ending battle and one that keeps us up at night continually coming up with new and better approaches, but one we relish (and, as they say, a nice problem to have).

Thanks for all the comments – it’s important for us to be challenged so keep them coming!

Craig

Dermott Renner
9 September 2009 #

Craig

Rackspace claim 100% uptime but they don’t achieve it as per a 45 minute downtime they had (and Xero had) 2 months ago. No one has 100% and it is naive for anyone to claim this. Just because their SLA says this doesn’t mean they or anyone else achieves.

Hosting web based systems with high availability in NZ is not impossible; we have done it for 4.5 years.

Datacom has a $30 million dollar data centre in Albany with Multi everything.

As far as “lots” of your customers having NZ, Australian and UK accounts I find that hard to believe. Must of your customers would be in NZ surely.

The single user account concept is not as important as reasonable speed.

Paul Rushworth
9 September 2009 #

Brynn,

An EC2 node in NZ? That’s optimism!

Dermott,

Hosting with Rackspace is just different to any other provider in the marketplace.

I can call and speak immediately to someone at rackspace 24/7/365 – and no, not a first level tech, but a system engineer, be it a member of the Virtualisation team, Storage, Backup, Network Security or Hardware, any time, day or night.

That is a fundamental difference between hosting with Rackspace and other hosting providers- people are not on-call- they’re onsite.

Jack
9 September 2009 #

I understand where you’re coming from with alot of your comments, but I side with the “Host some in NZ” team.

“What if you have a organization in NZ, Australia and the UK (which a lot of our customers do)? What if you travel? Do you expect your data to move with you on that flight to the UK?”
What if you were to let me choose where I host my data? If I were to be spending alot of time offshore I would choose to host offshore, and deal with the latency when in NZ. Since I spend 99.9% of my time in NZ, I would opt to have all the data hosted in NZ, and deal with it being in NZ while I’m overseas.

You also imply that hosting in NZ would result in more downtime, I would gladly trade a bit more downtime in return for Xero being alot more snappy. If you counted the seconds I spend waiting for pages to load that adds up for alot of downtime over a year! (Especially when I’m in Xero all day ;) )

James
10 September 2009 #

It is frustrating that this is such an issue for NZ companies.
Working out how to distribute databases in different regions has nothing to do with our business and is a complex problem. The day we solve it some company will lay a few cables across the Pacific Ocean and we will be maintaining an unnecessary complex mess.
International bandwidth is the problem. More international bandwidth is required to stop NZ companies wasting time on working around this issue and save NZ customers wasting time using slow internationally hosted products.
ps. I realise I am stating the obvious and not really contributing to the debate.

Ben Kepes
10 September 2009 #

James – you’re not stating the obvious

1) Latency – sure light travels fast but a NZ customer will always have a slower experience using a US hosted site – especially if there are multiple round trips (as an aside – check out aptimize for some cool web efficiency gains)
2) Jurisdiction is a major issue here. Already the Government is demanding that services it procures are domiciled locally. While not an issue for SMEs, it’s something we can’t ignore. Imagine, for example, the governemnt deciding that getting NZ SMEs onto SaaS apps was a good investment. Right now, if they were to procure those services themselves, companies like Xero would be unable to pitch for that contract…..

More here – https://www.cloudave.com/link/where-is-your-data

John Birse
12 September 2009 #

I would like to congratulate Xero for their transparency and willingness to discuss issues such as hosting openly. I couldn’t imagine other software suppliers being so open and this says heaps about Xero’s ethos and intent to value user opinion.

Trevor
28 October 2010 #

Speed is a critical factor for us. We are considering a shift to Xero and we are larger than ‘small’ with 140 staff and 9 campuses. Are there similar size organisations on Xero who can speak objectively about the speed of Xero in data input and particularly in report writing? We have outgrown QuickBoooks and wonder if Xero could cope?

Add your comment





We welcome all feedback but prefer a real name and email address.