Xero Blog



« | »

Infrastructure upgrade ensures resiliency

When you are growing a business as rapidly as Xero, doubling customer numbers year on year, and offering a service critical to so many people, you’ve got think well ahead. So we’re building and testing a new hosting environment for Xero that will ensure continued resilience and uptime and cope with this continual expansion.

We expect the server hardware and software at the base of Xero will need to handle a million+ customers in a few years time – that’s massive. Our existing hosting environment has been in place since late 2009 and has scaled from 15,000 paying customers to over 111,000.

We decided last year that a full migration to a new hosting environment is the appropriate next step rather than a further incremental capacity upgrade. This has allowed us to re-architect our platform to take advantage of new technologies, such as Microsoft SQL Server 2012, and scale in building blocks for easier planning, capacity and performance management.

As you would expect we have performed a considerable amount of testing and multiple trial migrations. We have rollback plans in place in the unlikely event the migration is not successful and we need to return to the current hosting environment.

The migration is scheduled for Sunday 27 Jan 8am to 4pm NZT (click here for your local time). The migration has a number of steps including final testing of the new hosting environment. We will bring Xero back online as soon as we are comfortable that it is ready, which may be earlier than 4pm.

UPDATE (2am 27 Jan UTC):  We have come back online thanks for your patience.

Posted by on 17 January 2013.

Categories: Company News

57 Responses

  1. Good work guys.

    I’m wondering, does this infrastructure upgrade solve the soft invoice limit of about 500 a month?

    by Wayne Robinson on Jan 17, 2013 at 11:21 am

  2. Hi Wayne,

    It’s really a design issue. 500 invoices per month and you’re in the realms of a separate invoicing sub ledger and sending daily summaries as journals through the API.

    It’s getting to an outlier of the market we’re targeting.

    Over time we might look to handle larger invoice numbers or even have an invoicing sub ledger product but that’s a ways away.

    Right now this is a good ‘apportunity’ for an add-on. I’d say it’s worth $499 to $999 a month for high volume invoicing. I know of alarm monitoring companies and even small ISPs looking for a volume billing solution that connects with Xero.

    Cheers

    Rod

    by Rod Drury on Jan 17, 2013 at 12:34 pm

  3. Will we notice any immediate differences after the migration?

    Hope all goes to plan for you on the 27th!

    by Jonathan Rata on Jan 17, 2013 at 12:44 pm

  4. @Jonathan – You shouldn’t notice any differences. The changes we are making are all under the hood.

    by Duncan Ritchie on Jan 17, 2013 at 1:23 pm

  5. Good luck Duncan, to you and the team!! Huge effort from the team and it will go well.

    by Christo Verster on Jan 17, 2013 at 4:16 pm

  6. @Rod Are you thinking something along the lines of a jBilling.com ?

    by Anthony on Jan 17, 2013 at 11:02 pm

  7. We issue about 800 invoices a month and are growing fast – I don’t know that we would pay $500 a month to keep using Xero but we would certainly pay *something* to be able to keep keep using it.

    But its really going to become a major problem for us soon. Nobody told us that giving up using our favourite product would be the price of success!

    by Matthew on Jan 23, 2013 at 8:17 pm

  8. What are you thinking?! In the UK it is the busiest time of year for accountants due to self assessment on 31st January many will be working around the clock.

    by Oliver Romain on Jan 24, 2013 at 12:59 am

  9. Just out of interest. Are you moving to Windows Azure? (I understand that you are on a MSFT stack and Azure would be a good candidate for the scale you need.)

    by Philip on Jan 24, 2013 at 1:14 am

  10. Hi Oliver – we’re aware of that and scheduled the downtime for 7pm on Saturday night until 3am Sunday morning UK time for that reason.

    by Gary Turner on Jan 24, 2013 at 1:42 am

  11. You guys need to seriously upgrade this programme:
    1) Logo’s on invoice – its so limiting the space you give – let us put whatever logo shape we want
    2) User profiles – i want sales resources to raise invoices but there is no user restriction to prevent them looking at sensitive reports or costs
    3) Reconciliation is a mess
    4) Search by amount on awaiting payment lists

    I could go on but for now that will do

    Thanks

    by Andrew on Jan 24, 2013 at 3:52 am

  12. I can’t believe it. The timing for the UK could not be worse.

    The personal tax return deadline is 31st January and it’s the busiest and most stressfuly time of year for accounts.

    Please postpone by a week!

    by Rex on Jan 24, 2013 at 4:37 am

  13. Is Xero considering having a ‘time billing’ module included into it – not an add-on, but as an ‘inclusion’? It would be very handy!

    by Marion on Jan 24, 2013 at 7:21 am

  14. Andrew, are you aware that you can indeed have any shape or size logo you want by using the customised .docx style of invoice? The possibilities are limitless with that method.

    by Chris on Jan 24, 2013 at 10:31 am

  15. YAY! I’m hoping the system is going to speed up with this change???? So hopefully there WILL be a noticeable change going forward? At the moment, it takes about 5 seconds for me to switch from my list of Repeating Invoices back to All Invoices. It feels like watching grass grow. :)

    by Karen on Jan 24, 2013 at 12:09 pm

  16. A high volume invoicing sub ledger add-on would be awesome.

    by Carl Aiau on Jan 24, 2013 at 1:19 pm

  17. No please don’t postpone by a week as its a 2 month GST Return filing starting 1st Feb. It seems like its down overnight in the UK, it means you need to stop work at 7pm Saturday and can re start in the early hours of Sunday morning.

    by Lyndsay on Jan 24, 2013 at 2:00 pm

  18. Well done Team Xero on being so transparent and open with the infrastructure that you have built your awesome product and business on. As someone who build SASS products, I aspire to be like “Xero”. And good on you Rod for listening and conversing with your customers. A real refreshing transition back to having that personal connection to your customers in today’s digital world.

    Good Luck with the Upgrade. I hope all goes smoothly and all those UK accounts can wake to sounds of birds chirping and Xero servers happily counting their beans. :)

    Jason

    by Jason Liddiard on Jan 24, 2013 at 6:20 pm

  19. Well done guys. Good to know thy it isn’t just all the pretty stuff you worry about. Sure we’ll all feel the difference though.

    With regards to timings of upgrade, well done. Balancing the demands of a global user base against the necessary infrastructure upgrades was no doubt a tough decision. And yes us UK accountants are working quite hard this time of year to tidy up the stragglers for tax returns.

    Saturday night UK time IS the compromise. Thank you Gary and keep up the good work.

    by Richard Adams on Jan 24, 2013 at 9:00 pm

  20. Wondering if you are considering making more merge fields available in the email templates. Would be great to have the [First Name] option so email greetings can be addressed to the client rather than the company… currently Hi [Company Name] … It would be great to say Hi [First Name]… any chance this may happen?

    by Malcolm on Jan 24, 2013 at 11:42 pm

  21. Being a developer/hosting provider for a small SaaS website I’m always interested to learn about a companies infrastructure – have you got any info on yours to share?

    by Ryan on Jan 25, 2013 at 12:40 am

  22. @Andrew. It looks like another user has answered your question about logos and has suggested using docx templates which is a good solution.

    Your other feature requests will be covered on our forum so I’d suggest that you head to this page https://community.xero.com/business/topic/23 and add your vote to the features you think are most important.

    by Duncan Ritchie on Jan 25, 2013 at 9:50 am

  23. @Marion and @Malcolm the best option is to place votes for the features you would like on our forum. Head over to https://community.xero.com/business/topic/23 and add your vote to the features you think are most important.

    by Duncan Ritchie on Jan 25, 2013 at 9:52 am

  24. @Ryan – We will be following up with some more detail after we go live.

    by Duncan Ritchie on Jan 25, 2013 at 9:52 am

  25. Has this migration begun already as we can’t get on today? I thought it was meant to begin on the weekend?

    by Ross Gales on Jan 25, 2013 at 1:08 pm

  26. @Ross apologies this wasn’t responded to on Friday. The migration begins today Sunday. If you have any issues logging into Xero outside the time span above please contact customer service at support@xero.com

    by Richard Wood on Jan 27, 2013 at 7:05 am

  27. Agree timing in the UK could not be worse!! Please consider deadlines in countries when doing such upgrades.

    by Anne Gallagher on Jan 27, 2013 at 7:16 am

  28. Great timing guys. We’re all working to a once-in-a-year (tax submission) deadline and you time your once-in-a-few-years upgrade to coincide with it. Brilliant! FYI, UK tax year ends 5-Apr and tax forms have to be in by 31-Jan. So, effectively, January is a no-go zone. Hopefully you’ll get a chance to fix user profile permissions so that we can allow access to basic information to key employees without giving them full access.

    by Greg on Jan 27, 2013 at 8:23 am

  29. Great job on this upgrade, Rod & team. There’s always going to be some transition pain and downtime during maintenance, but you need to ensure the infrastructure is scalable.

    by Dan Belhassen on Jan 27, 2013 at 9:25 am

  30. As the Director of an Ecommerce I’m on the verge of converting from Sage to Xero at the end of February, but was completely unaware of a limitation on Invoice numbers.

    What is the point of having integration with Ecommerce providers when we might be expecting daily invoices numbers to exceed 100.

    Please tell this isn’t true

    by Roni on Jan 27, 2013 at 9:27 am

  31. The people at Xero probably didn’t realise that most people’s financial year end 31st March and that they only had 9 months to get their accounts sorted. I used to be one of those people but then I got Xero!

    by Steve Toze on Jan 27, 2013 at 9:40 am

  32. Can u tell me when this will be up and running again New Zealand time please .

    by Natalie on Jan 27, 2013 at 10:09 am

  33. Yes we review all of the important dates across the countries we support however there are very few ideal times due to the global nature of our business. We scheduled for night in the UK to avoid times that people are working. Our apologies for interrupting your late Saturday work.

    by Duncan Ritchie on Jan 27, 2013 at 10:14 am

  34. @Natalie – The outage is scheduled through to 4pm NZT.

    by Duncan Ritchie on Jan 27, 2013 at 10:15 am

  35. Wow I didn’t realize you all were on an MSFT stack. That could explain some things, haha. How on earth did you all choose that as your platform of choice?

    by Andy on Jan 27, 2013 at 10:39 am

  36. Thank you for caring enough to upgrade the system. There will never be a perfect time to do it as everyone is in a different time zone. I wanted to get some work done here in Canada, but it can wait until tomorrow morning! To those in the UK, there could be worse inconveniences, Go enjoy a drink tonight and be fresh and ready to go in the morning.

    by Allison on Jan 27, 2013 at 11:37 am

  37. Huh? You have users in Seattle too with a Jan 31 deadline. If this is a worldwide tax thing (us & uk in these comments) why wasn’t update on feb 1st or later. I’m in a sort of bad spot now.

    by Sean on Jan 27, 2013 at 12:04 pm

  38. I’ll be ok, though.

    by Sean on Jan 27, 2013 at 12:06 pm

  39. @Sean – sorry I was responding the the UK comments. We are aware of the US deadlines as well. A big part of this migration is the ability to make changes without needing as many outages which should reduce the impact in the future.

    by Duncan Ritchie on Jan 27, 2013 at 12:09 pm

  40. You cannot be serious. The 28th is when all GST and super payments are due to the ATO in Australia. most of my accounting reconciliation is complete but I need to check and reconcile my BAS and pay my employees super. Today. I did not do it yesterday as it is my one day off a week knowing I could hook into Xero today. Not. Apparently. Dumb timing for a service we pay for. unbelievable………

    by Mark Semmens on Jan 27, 2013 at 1:10 pm

  41. Like expense claims, I want to attach PDFs or jpgs to any or all transactions. Is this a capacity issue for Xero?

    by Stewart on Jan 27, 2013 at 1:47 pm

  42. Yay Xero is true to their word we are back up and running 1/2 hr before eta

    by Gillian Rossouw on Jan 27, 2013 at 3:34 pm

  43. from today I have definitely noticed a slower response time when processing new invoices. Will this improves when your new hardware gets embedded a bit more?

    Located. Wellington.

    by Andrew on Jan 27, 2013 at 4:10 pm

  44. @Andrew – Some of the processes take a little while to warm up to normal usage patterns. This should happen reasonably quickly and our monitoring indicates that the system is performing well.

    by Duncan Ritchie on Jan 27, 2013 at 4:21 pm

  45. yer, its got a little quicker in the last 30 minutes. Sounds pedantic, but when your processing lots of invoices any delay can be a pain in the arse.

    by Andrew on Jan 27, 2013 at 4:45 pm

  46. @roni, if you’re doing hundreds of invoices per day then your processes would probably we better with a dedicated subsidiary ledger for invoicing with appropriate automation to handle those volumes and only send daily batch summaries through the API into accounting.

    We have considered a more ‘headless’ high volume invoicing system but high volumes of invoices are outliers so hasn’t been a priority yet. At our price point and life cycle we have to chase the volume market.

    Higher volume invoicing is a scenario we’d like to handle in the future but we think it’s best to be up front that we have not yet designed or tuned invoicing for several thousand per month.

    Rod

    by Rod Drury on Jan 27, 2013 at 5:05 pm

  47. Xero dudes. Can you set up a uservoice for general wishlist items for the Xero UI and features?

    by Sean Hoyt on Jan 27, 2013 at 9:20 pm

  48. @Sean – Check out our community it has a feature request section with voting. https://community.xero.com/business/topic/23

    by Duncan Ritchie on Jan 27, 2013 at 9:58 pm

  49. Wouldn’t a migration to an “elastic compute” platform, like Azure, resolve future capacity needs and mean you don’t have to switch off in future?

    by Adrian Pearson on Jan 27, 2013 at 10:16 pm

  50. As I right this, reports are taking along time to run at the moment. Also, report exports to Google Docs or Excel are failing intermittently.

    Is the Xero Ops team aware of such issues?

    by Jev Bjorsell on Jan 28, 2013 at 9:45 am

  51. @Jev – We have double checked and there are no system wide issues. Can you log a support ticket and we will dig deeper for you?

    by Duncan Ritchie on Jan 28, 2013 at 12:11 pm

  52. I can sign in and work normally, but when I try to open an online invoice from a supplier it takes me to the “offline for maintenance” page…?

    by Dominic on Jan 28, 2013 at 4:40 pm

  53. @Dominic – I’ve just had a couple of people test this and they haven’t had any issues. Can you log a support ticket and they will help you resolve the issue. support@xero.com

    by Duncan Ritchie on Jan 28, 2013 at 7:44 pm

  54. Rod, is that 500 invoice ‘soft’ limit still in place?

    I use Xero for an active e-commerce site. We have 5,000+ monthly PayPal orders. We’re not big – most of these sales are for a few bucks each… but we’re going to be relying on Xero’s PayPal integration to account for them.

    Ideally, I’d like to use Xero’s API to send every customer a VAT invoice, but it sounds like you’re not set-up to do this?

    Happy to pay more than the $25/mo to achieve that. Not sure it’s a $499-999 leap for what is essentially an existing feature scaled out, but a bump to a $99-$199/mo package would be more than reasonable.

    Also – I’d really like to learn more about how you’re currently achieving your scale. How you’ve managed to stick with Microsoft and achieve a 1M active user scale, I’d love to know! Our ASP/SQL servers needed rebooting every other week when we hit 100k users (another business). We wound up taking a Node.js/NoSQL route on barebones Linux… and we were a fraction of your scale!

    You should feature that achievement somewhere – kudos for keeping up with the growth, I know it’s not easy!

    by Lee on Apr 4, 2013 at 8:41 pm

  55. @Lee

    1. Yes we have designed the UI and tuned for 99% of users which is well under 500 invoices.
    2. Will leave that question to our tech team. I do know we run shards so we scale out by adding servers horizontally. We do have complex cross shard views that make this non trivial but that extinguishes my jargon quota.

    by Rod Drury on Apr 4, 2013 at 10:41 pm

  56. @Lee .NET actually scales really well – it’s extremely fast and well-tuned, especially on Windows. SQL Server is also extremely powerful (especially SQL Enterprise).

    However at some point scaling up (which is the standard Microsoft way) is not feasible – it just costs too much money. Therefore we horizontally partitioned (sharded) early – so early, in fact, that for a while we were only running 1 shard. But the important thing is that because we sharded and built it into our architecture, we were able to scale out very quickly. So now we run x shards on y database servers (each instance of SQL Server runs a few shards – rebalancing can be as simple as moving a shard to it’s own instance).

    Even with a Node/NoSQL route you will need to scale out at some point (in fact if you’re running Node you’re probably scaling out already – a Node process can only target one core so easy Node scaling is simply targeting more cores using something like cluster).

    Having said all that, for the amount of data we have, and the number of customers we have to service, we do it on a relatively small amount of hardware. A lot of that has to do with our choice of .NET.

    by Craig Walker on Apr 5, 2013 at 6:29 am

  57. Hi, i’m still thinking shall i join Xero. Am using the trial version at the moment, wondering what will happen when i hit 100 and more invoices? Will my account be blocked or I cannot send out invoice by this system?

    I’m looking forward to your favorable reply. Many thanks!

    by CatindaHat on Apr 28, 2013 at 5:44 am

Leave a Reply

« | »




Recent Posts


Pages