Skip to content

Infrastructure upgrade ensures resiliency

When you are growing a business as rapidly as Xero, doubling customer numbers year on year, and offering a service critical to so many people, you’ve got think well ahead. So we’re building and testing a new hosting environment for Xero that will ensure continued resilience and uptime and cope with this continual expansion.

We expect the server hardware and software at the base of Xero will need to handle a million+ customers in a few years time – that’s massive. Our existing hosting environment has been in place since late 2009 and has scaled from 15,000 paying customers to over 111,000.

We decided last year that a full migration to a new hosting environment is the appropriate next step rather than a further incremental capacity upgrade. This has allowed us to re-architect our platform to take advantage of new technologies, such as Microsoft SQL Server 2012, and scale in building blocks for easier planning, capacity and performance management.

As you would expect we have performed a considerable amount of testing and multiple trial migrations. We have rollback plans in place in the unlikely event the migration is not successful and we need to return to the current hosting environment.

The migration is scheduled for Sunday 27 Jan 8am to 4pm NZT (click here for your local time). The migration has a number of steps including final testing of the new hosting environment. We will bring Xero back online as soon as we are comfortable that it is ready, which may be earlier than 4pm.

UPDATE (2am 27 Jan UTC):  We have come back online thanks for your patience.

 

Read more about Company News

 

57 comments

Wayne Robinson
17 January 2013 #

Good work guys.

I’m wondering, does this infrastructure upgrade solve the soft invoice limit of about 500 a month?

Rod Drury
17 January 2013 #

Hi Wayne,

It’s really a design issue. 500 invoices per month and you’re in the realms of a separate invoicing sub ledger and sending daily summaries as journals through the API.

It’s getting to an outlier of the market we’re targeting.

Over time we might look to handle larger invoice numbers or even have an invoicing sub ledger product but that’s a ways away.

Right now this is a good ‘apportunity’ for an add-on. I’d say it’s worth $499 to $999 a month for high volume invoicing. I know of alarm monitoring companies and even small ISPs looking for a volume billing solution that connects with Xero.

Cheers

Rod

Jonathan Rata
17 January 2013 #

Will we notice any immediate differences after the migration?

Hope all goes to plan for you on the 27th!

Duncan Ritchie
17 January 2013 #

@Jonathan – You shouldn’t notice any differences. The changes we are making are all under the hood.

Christo Verster
17 January 2013 #

Good luck Duncan, to you and the team!! Huge effort from the team and it will go well.

Anthony
17 January 2013 #

@Rod Are you thinking something along the lines of a jBilling.com ?

Matthew
23 January 2013 #

We issue about 800 invoices a month and are growing fast – I don’t know that we would pay $500 a month to keep using Xero but we would certainly pay *something* to be able to keep keep using it.

But its really going to become a major problem for us soon. Nobody told us that giving up using our favourite product would be the price of success!

Oliver Romain
24 January 2013 #

What are you thinking?! In the UK it is the busiest time of year for accountants due to self assessment on 31st January many will be working around the clock.

Philip
24 January 2013 #

Just out of interest. Are you moving to Windows Azure? (I understand that you are on a MSFT stack and Azure would be a good candidate for the scale you need.)

Gary Turner
24 January 2013 #

Hi Oliver – we’re aware of that and scheduled the downtime for 7pm on Saturday night until 3am Sunday morning UK time for that reason.

Andrew
24 January 2013 #

You guys need to seriously upgrade this programme:
1) Logo’s on invoice – its so limiting the space you give – let us put whatever logo shape we want
2) User profiles – i want sales resources to raise invoices but there is no user restriction to prevent them looking at sensitive reports or costs
3) Reconciliation is a mess
4) Search by amount on awaiting payment lists

I could go on but for now that will do

Thanks

Rex
24 January 2013 #

I can’t believe it. The timing for the UK could not be worse.

The personal tax return deadline is 31st January and it’s the busiest and most stressfuly time of year for accounts.

Please postpone by a week!

Marion
24 January 2013 #

Is Xero considering having a ‘time billing’ module included into it – not an add-on, but as an ‘inclusion’? It would be very handy!

Chris
24 January 2013 #

Andrew, are you aware that you can indeed have any shape or size logo you want by using the customised .docx style of invoice? The possibilities are limitless with that method.

Karen
24 January 2013 #

YAY! I’m hoping the system is going to speed up with this change???? So hopefully there WILL be a noticeable change going forward? At the moment, it takes about 5 seconds for me to switch from my list of Repeating Invoices back to All Invoices. It feels like watching grass grow. :)

Carl Aiau
24 January 2013 #

A high volume invoicing sub ledger add-on would be awesome.

Lyndsay
24 January 2013 #

No please don’t postpone by a week as its a 2 month GST Return filing starting 1st Feb. It seems like its down overnight in the UK, it means you need to stop work at 7pm Saturday and can re start in the early hours of Sunday morning.

Jason Liddiard
24 January 2013 #

Well done Team Xero on being so transparent and open with the infrastructure that you have built your awesome product and business on. As someone who build SASS products, I aspire to be like “Xero”. And good on you Rod for listening and conversing with your customers. A real refreshing transition back to having that personal connection to your customers in today’s digital world.

Good Luck with the Upgrade. I hope all goes smoothly and all those UK accounts can wake to sounds of birds chirping and Xero servers happily counting their beans. :)

Jason

Richard Adams
24 January 2013 #

Well done guys. Good to know thy it isn’t just all the pretty stuff you worry about. Sure we’ll all feel the difference though.

With regards to timings of upgrade, well done. Balancing the demands of a global user base against the necessary infrastructure upgrades was no doubt a tough decision. And yes us UK accountants are working quite hard this time of year to tidy up the stragglers for tax returns.

Saturday night UK time IS the compromise. Thank you Gary and keep up the good work.

Malcolm
24 January 2013 #

Wondering if you are considering making more merge fields available in the email templates. Would be great to have the [First Name] option so email greetings can be addressed to the client rather than the company… currently Hi [Company Name] … It would be great to say Hi [First Name]… any chance this may happen?

Ryan
25 January 2013 #

Being a developer/hosting provider for a small SaaS website I’m always interested to learn about a companies infrastructure – have you got any info on yours to share?

Duncan Ritchie
25 January 2013 #

@Andrew. It looks like another user has answered your question about logos and has suggested using docx templates which is a good solution.

Your other feature requests will be covered on our forum so I’d suggest that you head to this page https://community.xero.com/business/topic/23 and add your vote to the features you think are most important.

Duncan Ritchie
25 January 2013 #

@Marion and @Malcolm the best option is to place votes for the features you would like on our forum. Head over to https://community.xero.com/business/topic/23 and add your vote to the features you think are most important.

Duncan Ritchie
25 January 2013 #

@Ryan – We will be following up with some more detail after we go live.

Ross Gales
25 January 2013 #

Has this migration begun already as we can’t get on today? I thought it was meant to begin on the weekend?

Richard Wood
27 January 2013 #

@Ross apologies this wasn’t responded to on Friday. The migration begins today Sunday. If you have any issues logging into Xero outside the time span above please contact customer service at support@xero.com

Anne Gallagher
27 January 2013 #

Agree timing in the UK could not be worse!! Please consider deadlines in countries when doing such upgrades.

Greg
27 January 2013 #

Great timing guys. We’re all working to a once-in-a-year (tax submission) deadline and you time your once-in-a-few-years upgrade to coincide with it. Brilliant! FYI, UK tax year ends 5-Apr and tax forms have to be in by 31-Jan. So, effectively, January is a no-go zone. Hopefully you’ll get a chance to fix user profile permissions so that we can allow access to basic information to key employees without giving them full access.

Dan Belhassen
27 January 2013 #

Great job on this upgrade, Rod & team. There’s always going to be some transition pain and downtime during maintenance, but you need to ensure the infrastructure is scalable.

Roni
27 January 2013 #

As the Director of an Ecommerce I’m on the verge of converting from Sage to Xero at the end of February, but was completely unaware of a limitation on Invoice numbers.

What is the point of having integration with Ecommerce providers when we might be expecting daily invoices numbers to exceed 100.

Please tell this isn’t true

Steve Toze
27 January 2013 #

The people at Xero probably didn’t realise that most people’s financial year end 31st March and that they only had 9 months to get their accounts sorted. I used to be one of those people but then I got Xero!

Natalie
27 January 2013 #

Can u tell me when this will be up and running again New Zealand time please .

Duncan Ritchie
27 January 2013 #

Yes we review all of the important dates across the countries we support however there are very few ideal times due to the global nature of our business. We scheduled for night in the UK to avoid times that people are working. Our apologies for interrupting your late Saturday work.

Duncan Ritchie
27 January 2013 #

@Natalie – The outage is scheduled through to 4pm NZT.

Andy
27 January 2013 #

Wow I didn’t realize you all were on an MSFT stack. That could explain some things, haha. How on earth did you all choose that as your platform of choice?

Allison
27 January 2013 #

Thank you for caring enough to upgrade the system. There will never be a perfect time to do it as everyone is in a different time zone. I wanted to get some work done here in Canada, but it can wait until tomorrow morning! To those in the UK, there could be worse inconveniences, Go enjoy a drink tonight and be fresh and ready to go in the morning.

Sean
27 January 2013 #

Huh? You have users in Seattle too with a Jan 31 deadline. If this is a worldwide tax thing (us & uk in these comments) why wasn’t update on feb 1st or later. I’m in a sort of bad spot now.

Sean
27 January 2013 #

I’ll be ok, though.

Duncan Ritchie
27 January 2013 #

@Sean – sorry I was responding the the UK comments. We are aware of the US deadlines as well. A big part of this migration is the ability to make changes without needing as many outages which should reduce the impact in the future.

Mark Semmens
27 January 2013 #

You cannot be serious. The 28th is when all GST and super payments are due to the ATO in Australia. most of my accounting reconciliation is complete but I need to check and reconcile my BAS and pay my employees super. Today. I did not do it yesterday as it is my one day off a week knowing I could hook into Xero today. Not. Apparently. Dumb timing for a service we pay for. unbelievable………

Stewart
27 January 2013 #

Like expense claims, I want to attach PDFs or jpgs to any or all transactions. Is this a capacity issue for Xero?

Gillian Rossouw
27 January 2013 #

Yay Xero is true to their word we are back up and running 1/2 hr before eta

Andrew
27 January 2013 #

from today I have definitely noticed a slower response time when processing new invoices. Will this improves when your new hardware gets embedded a bit more?

Located. Wellington.

Duncan Ritchie
27 January 2013 #

@Andrew – Some of the processes take a little while to warm up to normal usage patterns. This should happen reasonably quickly and our monitoring indicates that the system is performing well.

Andrew
27 January 2013 #

yer, its got a little quicker in the last 30 minutes. Sounds pedantic, but when your processing lots of invoices any delay can be a pain in the arse.

Rod Drury
27 January 2013 #

@roni, if you’re doing hundreds of invoices per day then your processes would probably we better with a dedicated subsidiary ledger for invoicing with appropriate automation to handle those volumes and only send daily batch summaries through the API into accounting.

We have considered a more ‘headless’ high volume invoicing system but high volumes of invoices are outliers so hasn’t been a priority yet. At our price point and life cycle we have to chase the volume market.

Higher volume invoicing is a scenario we’d like to handle in the future but we think it’s best to be up front that we have not yet designed or tuned invoicing for several thousand per month.

Rod

Sean Hoyt
27 January 2013 #

Xero dudes. Can you set up a uservoice for general wishlist items for the Xero UI and features?

Duncan Ritchie
27 January 2013 #

@Sean – Check out our community it has a feature request section with voting. https://community.xero.com/business/topic/23

Adrian Pearson
27 January 2013 #

Wouldn’t a migration to an “elastic compute” platform, like Azure, resolve future capacity needs and mean you don’t have to switch off in future?

Jev Bjorsell
28 January 2013 #

As I right this, reports are taking along time to run at the moment. Also, report exports to Google Docs or Excel are failing intermittently.

Is the Xero Ops team aware of such issues?

Duncan Ritchie
28 January 2013 #

@Jev – We have double checked and there are no system wide issues. Can you log a support ticket and we will dig deeper for you?

Dominic
28 January 2013 #

I can sign in and work normally, but when I try to open an online invoice from a supplier it takes me to the “offline for maintenance” page…?

Duncan Ritchie
28 January 2013 #

@Dominic – I’ve just had a couple of people test this and they haven’t had any issues. Can you log a support ticket and they will help you resolve the issue. support@xero.com

Lee
4 April 2013 #

Rod, is that 500 invoice ‘soft’ limit still in place?

I use Xero for an active e-commerce site. We have 5,000+ monthly PayPal orders. We’re not big – most of these sales are for a few bucks each… but we’re going to be relying on Xero’s PayPal integration to account for them.

Ideally, I’d like to use Xero’s API to send every customer a VAT invoice, but it sounds like you’re not set-up to do this?

Happy to pay more than the $25/mo to achieve that. Not sure it’s a $499-999 leap for what is essentially an existing feature scaled out, but a bump to a $99-$199/mo package would be more than reasonable.

Also – I’d really like to learn more about how you’re currently achieving your scale. How you’ve managed to stick with Microsoft and achieve a 1M active user scale, I’d love to know! Our ASP/SQL servers needed rebooting every other week when we hit 100k users (another business). We wound up taking a Node.js/NoSQL route on barebones Linux… and we were a fraction of your scale!

You should feature that achievement somewhere – kudos for keeping up with the growth, I know it’s not easy!

Rod Drury
4 April 2013 #

@Lee

1. Yes we have designed the UI and tuned for 99% of users which is well under 500 invoices.
2. Will leave that question to our tech team. I do know we run shards so we scale out by adding servers horizontally. We do have complex cross shard views that make this non trivial but that extinguishes my jargon quota.

Craig Walker
5 April 2013 #

@Lee .NET actually scales really well – it’s extremely fast and well-tuned, especially on Windows. SQL Server is also extremely powerful (especially SQL Enterprise).

However at some point scaling up (which is the standard Microsoft way) is not feasible – it just costs too much money. Therefore we horizontally partitioned (sharded) early – so early, in fact, that for a while we were only running 1 shard. But the important thing is that because we sharded and built it into our architecture, we were able to scale out very quickly. So now we run x shards on y database servers (each instance of SQL Server runs a few shards – rebalancing can be as simple as moving a shard to it’s own instance).

Even with a Node/NoSQL route you will need to scale out at some point (in fact if you’re running Node you’re probably scaling out already – a Node process can only target one core so easy Node scaling is simply targeting more cores using something like cluster).

Having said all that, for the amount of data we have, and the number of customers we have to service, we do it on a relatively small amount of hardware. A lot of that has to do with our choice of .NET.

CatindaHat
28 April 2013 #

Hi, i’m still thinking shall i join Xero. Am using the trial version at the moment, wondering what will happen when i hit 100 and more invoices? Will my account be blocked or I cannot send out invoice by this system?

I’m looking forward to your favorable reply. Many thanks!

Add your comment





We welcome all feedback but prefer a real name and email address.