We are currently experiencing an issue within our application hosting infrastructure that is affecting a proportion of Xero Customers. Xero’s operations team is working to identify and resolve the cause of the issue an as soon as possible.
We will provide any updates, including expected time for the service to be restored, as soon as we have more information.
11:48 NZT: We’ve identified the issue and are currently working to get everyone back up and running. Thank you for your continued patience.
11:59 NZT: Our provider is working to fix their hardware. We’re getting closer to getting you back on Xero. We are sorry for the disruption this is causing you.
12:10 NZT: Our provider has identified and is replacing faulty hardware and we’re working on our end to get everyone back online.
12:18 NZT: We are sorry about the frustration this may be causing. We very rarely have extended outages like this and thank you for your support while we pull out all the stops to get this fixed.
12:33 NZT: The outage is now affecting all users. We cannot tell you how sorry we are about this interruption. We are working hard alongside our provider to get this resolved.
12:56 NZT: We can confirm that no data has been lost as a result of this outage.
1:11 NZT: We have resolved the issue. Please look forward to a debrief from us on the outage. We, again, are very sorry for the disruption this has caused you.
UPDATE
Xero was hit earlier today with an outage that affected many of our customers for approximately two hours. We’re closely monitoring our systems to ensure everything is healthy again.
We had our dedicated teams working on this issue to get you up and running again and the issue was resolved as quickly as possible. Know that we’re already working to ensure that this doesn’t happen again.
The incident was initially triggered by a hardware outage, which caused instability in our core network.
We don’t have a clear root cause yet. However, the likely reason is that it was a software issue (triggered by the hardware failure) in the switch that corrupted both the primary and redundant network infrastructure.
The way that redundancy works in these situations is that the redundant switches monitor each other and the network and decides when to fail safe over. The system was fully operational when it was last tested two weeks ago. Both the A and B sides of our network were able to run independently and the automated failover between them worked as expected. That didn’t happen this time.
We’re working with the equipment manufacturer and our provider to determine why both sides were impacted, but it’s important to note that there was no data lost and this is not a security issue.
We are migrating to the public cloud which will help ensure this doesn’t happen in the future. The way we will operate in the public cloud is quite different to our current infrastructure and while it removes the likelihood of similar issues, every environment has an element of outage risk.
We know what broke and we knew how to fix it, but we don’t know why it broke, yet. Our team is currently conducting a thorough post-mortem with the hardware manufacturer and our service provider to ensure that we understand the underlying root cause of the issue and better prepare for potential network outages in the future.
Our last major outage was more than five years ago and despite this morning’s outage, we have maintained an uptime of 99.97% for the last 12 months, and will strive to retain our industry-leading 99.99% uptime into the future.
We apologise for letting you down.
Thank you for keeping us informed through the process – appreciate it.
Check your log in people – I’m back in! Can’t see that Xero has updated yet.
Could be so much worse… could be using MYOB!
You took the words right out of my mouth!
Thanks for the update, earning no money is not ideal, but the swell is running, so Im off for a surf!
We have now resolved the issue. Everyone should be back up and running.
Thank you, much appreciated 🙂
Just wanted to say thanks for the communication throughout the morning. Great product, great service. Thanks very much.
Thanks. 🙂
Xero is up and running!!
Yeah Thank you its working now!!!!!
xero is back now. Thanks for the effort.
Thanks Xero for giving me the day off so I can watch Australia beat New Zealand in the cricket.
Please continue with you’re excellent updates and hopefully things will be up and running once the cricket is finished.
For everyone, else go take a walk, breath and be thankful for your good health.
Thanks to Xero for keeping us in the loop. Hardware failures happen. Nothing is infallible. Sure it’s inconvenient for all of us but they are doing their best to rectify it ASAP. I think it’s a bloody great product at very little cost to our company.
I think it’s important to maintain perspective at times like this.
Anyone remember when Quickbooks Online was down for 3 days in 2012 because someone tripped over an extension cord? talk about incompetence.
We’re back in business!
It’s working again for me too
Thanks for the update, we will await for the fix. Xero is great software.
Working now, it was only a tad fraught because it’s payday today and us in admin didn’t want to face a mutiny of staff. Some of them are big and kind of scary.
Good job to the techs!
Surely there should backup plans in place for these instances, it’s not Xero’s business that is down there are 1000’s of business that are down. All loosing money waiting… Not a happy customer.
keep drinking that cloud Koolaid. When it goes down/gets hacked MANY businesses are/will be effected -> not just one… anyone with basic IT experience can see the downsides of cloud computing.
YAY thanks so much up and running 🙂
Thanks for keeping us up to date and getting us back up and running. We’ve been moving clients to Xero since 2012 and this is the first time we’ve been unable to access the data at any time anywhere in the world so this is nothing compared to the old desktop days. Keep up the excellent work!
Thank you for the updates – and now you’re fixed – good job – 2 hours downtime in over 2 years of using Xero? – not bad!
My second time in two years – of not being able to access.
well done every one, you deserve a drink ?
Have you ever considered a back up plan? Unbelievable that you don’t have one.
They detail the backup plan in the blog post.
If you read their update they stated there was a plan but that failed too
Oh how quickly some people’s expectations have become totally unreasonable when it comes to technology. Obviously these people weren’t on the internet during dial-up times. If you can do it better, you go invent it. Go on.. shoo.
Well said James
Good effort Xero :):)
Thanks for keeping us in the loop and your timely resolution. Can’t imagine we would have such a quick response if our own service was down.
We just came back up. Thanks. Stressful time for us all, but first time in about 5 years, and if I’d been using an installed system it could even more likely be my own system that went down.
Just don’t let it happen again.
Can’t even watch cricket, now.
Thanks Xero, no casualties and life goes on
In the years that I’ve been working with Xero this is the first outage that has ever affected me. These things happen. However, you guys quickly replied to my email, kept me updated and got things running reasonably quickly. The communication was brilliant. Well done Xero!
thanks – have a nice day
Great work from your team Xero.
Unfortunately you can never 100% rule out outages – it still happens now and then for sites like Facebook, Twitter and YouTube. Unfortunately the proverbial happens, and while we can try and limit it, we can never prevent it completely.
The real measure of a quality software provider is in their response to such an outage. Xero has given us pretty good communication here, with the longest spell between updates being 23 minutes. On top of that, the issue was resolved relatively quickly (I’ve had other outages from other IT providers much longer than that), and most importantly of all, no data was lost.
Obviously a hassle for those who needed to do something right that minute, but definitely reassuring that these outages are rare, and well handled. Thanks team 🙂
You can spot all the people on here who leave all their work till the last minute. 🙂
I have been using Xero for some time now and this is the first time I can recall the system going down. thank you for getting things up and running as quickly as you did and also for keeping us informed through the process.
Well done guys – great transparency and a quick resolution. Keep up the excellent work.
I can’t log in…
let us all not forget that it was not so long ago that all the transactions had to be hand posted into a ledger book, using a fountain pen! From one who has done such work I assure you, nothing is infallible as what can go wrong will go wrong , Murphy’s Law.
Thanks for the updates, and happy to see service is now restored. After reading your update, I am confused with paragraph 7:
“We are migrating to the public cloud which will help ensure this doesn’t happen in the future. The way we will operate in the public cloud is quite different to our current infrastructure and while it removes the likelihood of similar issues, every environment has an element of outage risk.”
Can you please clarify the migration to the public cloud?
You are the best!
I forgive you 🙂
xoxox
Thanks for the information and regular updates. It’s nice to be kept in the loop. It is a pity other large companies don’t respond the same when they have issues. Telco’s in particular.
Can we get clarification on ‘moving to the cloud’ statement.
Thanks
Wow, talk about a thorough explanation…awesome…. just like your Rugby players! True professionals that’s what we like.
Lucky is wasn’t myob
They would have blamed the user
Well done people
I love Xero
So impressed with how you’ve kept us informed with your honesty. I’ve spent longer in phone queues to MYOB to hear lies!
You’re still beautiful software to me! 😉
Well done on your communication and keeping everybody informed along the way – made a huge difference. Every company will have times like this but its all about how you handle it. Good job – still love xero!
Thanks for managing to fix this problem so quickly – pretty amazing – & replying to our queries about it as well. Technology will never be perfect!
I am trying to complete the surveys from the Roadshow in Melbourne and the links provided result in an error message. Second day in a row with problems….oops!
Hi Peter – Sorry you’re having trouble with the link. The surveys were unaffected by the outage and are up and running. Give them a try here: https://www.etouches.com/esurvey/auxeroroadshowfeb2016
Has anyone else noticed bank statement not being uploaded in Xero this morning. We have three entities and none of the bank statements are uploaded. Banks we use are CBA and Westpac.
You can find all the information on bank feeds here: https://www.xero.com/blog/bank-feed-status/
It’s been terrible all morning today. Down, up down, up. What is going on???
You can find all the information on bank feeds here: https://www.xero.com/blog/bank-feed-status/
This is now getting to be a serious issue; not updated feeds since 23 Feb; causing work backlog and month end delays.Moving into Day 8
You can find all the information about Bank Feeds here: https://www.xero.com/blog/bank-feed-status/
If you’re still experiencing trouble you can contact customer support here: support@xero.com