Infrastructure upgrade ensures resiliency

When you are growing a business as rapidly as Xero, doubling customer numbers year on year, and offering a service critical to so many people, you’ve got think well ahead. So we’re building and testing a new hosting environment for Xero that will ensure continued resilience and uptime and cope with this continual expansion.

We expect the server hardware and software at the base of Xero will need to handle a million+ customers in a few years time – that’s massive. Our existing hosting environment has been in place since late 2009 and has scaled from 15,000 paying customers to over 111,000.

We decided last year that a full migration to a new hosting environment is the appropriate next step rather than a further incremental capacity upgrade. This has allowed us to re-architect our platform to take advantage of new technologies, such as Microsoft SQL Server 2012, and scale in building blocks for easier planning, capacity and performance management.

As you would expect we have performed a considerable amount of testing and multiple trial migrations. We have rollback plans in place in the unlikely event the migration is not successful and we need to return to the current hosting environment.

The migration is scheduled for Sunday 27 Jan 8am to 4pm NZT (click here for your local time). The migration has a number of steps including final testing of the new hosting environment. We will bring Xero back online as soon as we are comfortable that it is ready, which may be earlier than 4pm.

UPDATE (2am 27 Jan UTC):  We have come back online thanks for your patience.

57 Comments

Duncan Ritchie
January 28, 2013 at 12:11 pm

@Jev – We have double checked and there are no system wide issues. Can you log a support ticket and we will dig deeper for you?

Dominic
January 28, 2013 at 4:40 pm

I can sign in and work normally, but when I try to open an online invoice from a supplier it takes me to the “offline for maintenance” page…?

Duncan Ritchie
January 28, 2013 at 7:44 pm

@Dominic – I’ve just had a couple of people test this and they haven’t had any issues. Can you log a support ticket and they will help you resolve the issue. support@xero.com

Lee
April 4, 2013 at 8:41 pm

Rod, is that 500 invoice ‘soft’ limit still in place?

I use Xero for an active e-commerce site. We have 5,000+ monthly PayPal orders. We’re not big – most of these sales are for a few bucks each… but we’re going to be relying on Xero’s PayPal integration to account for them.

Ideally, I’d like to use Xero’s API to send every customer a VAT invoice, but it sounds like you’re not set-up to do this?

Happy to pay more than the $25/mo to achieve that. Not sure it’s a $499-999 leap for what is essentially an existing feature scaled out, but a bump to a $99-$199/mo package would be more than reasonable.

Also – I’d really like to learn more about how you’re currently achieving your scale. How you’ve managed to stick with Microsoft and achieve a 1M active user scale, I’d love to know! Our ASP/SQL servers needed rebooting every other week when we hit 100k users (another business). We wound up taking a Node.js/NoSQL route on barebones Linux… and we were a fraction of your scale!

You should feature that achievement somewhere – kudos for keeping up with the growth, I know it’s not easy!

Rod Drury
April 4, 2013 at 10:41 pm

@Lee

1. Yes we have designed the UI and tuned for 99% of users which is well under 500 invoices.
2. Will leave that question to our tech team. I do know we run shards so we scale out by adding servers horizontally. We do have complex cross shard views that make this non trivial but that extinguishes my jargon quota.

Craig Walker
April 5, 2013 at 6:29 am

@Lee .NET actually scales really well – it’s extremely fast and well-tuned, especially on Windows. SQL Server is also extremely powerful (especially SQL Enterprise).

However at some point scaling up (which is the standard Microsoft way) is not feasible – it just costs too much money. Therefore we horizontally partitioned (sharded) early – so early, in fact, that for a while we were only running 1 shard. But the important thing is that because we sharded and built it into our architecture, we were able to scale out very quickly. So now we run x shards on y database servers (each instance of SQL Server runs a few shards – rebalancing can be as simple as moving a shard to it’s own instance).

Even with a Node/NoSQL route you will need to scale out at some point (in fact if you’re running Node you’re probably scaling out already – a Node process can only target one core so easy Node scaling is simply targeting more cores using something like cluster).

Having said all that, for the amount of data we have, and the number of customers we have to service, we do it on a relatively small amount of hardware. A lot of that has to do with our choice of .NET.

CatindaHat
April 28, 2013 at 5:44 am

Hi, i’m still thinking shall i join Xero. Am using the trial version at the moment, wondering what will happen when i hit 100 and more invoices? Will my account be blocked or I cannot send out invoice by this system?

I’m looking forward to your favorable reply. Many thanks!

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

Xero staff and partners star in the Xero Band

It’s been two weeks since Xerocon Melbourne and we’re loving all the great feedback that’s come through. It’s fantastic to hear what everyone loved the most. Thanks so much to everyone who attended and helped make it happen! The Xero Band rocks for the fourth year running As always, one of the highlights was the ...

New IDC report shows Xero is a global growth leader

Xero is the fastest growing financial applications vendor in the world according to the International Data Corporation’s new report. IDC, a world leader in technology research, recently released their annual report. They report on market sizing and share for finance and financial payroll software providers*. Not only was Xero the fastest growing financial application, it ...