Brought to you by

Unexpected outage for Xero customers (RESOLVED)

Posted 2 years ago in Xero news by Duncan Ritchie
Posted by Duncan Ritchie

We are currently experiencing an issue within our application hosting infrastructure that is affecting a proportion of Xero Customers. Xero’s operations team is working to identify and resolve the cause of the issue an as soon as possible.

We will provide any updates, including expected time for the service to be restored, as soon as we have more information.

11:48 NZT: We’ve identified the issue and are currently working to get everyone back up and running. Thank you for your continued patience.

11:59 NZT: Our provider is working to fix their hardware. We’re getting closer to getting you back on Xero. We are sorry for the disruption this is causing you.

12:10 NZT: Our provider has identified and is replacing faulty hardware and we’re working on our end to get everyone back online.

12:18 NZT: We are sorry about the frustration this may be causing. We very rarely have extended outages like this and thank you for your support while we pull out all the stops to get this fixed.

12:33 NZT: The outage is now affecting all users. We cannot tell you how sorry we are about this interruption. We are working hard alongside our provider to get this resolved.

12:56 NZT: We can confirm that no data has been lost as a result of this outage.

1:11 NZT: We have resolved the issue. Please look forward to a debrief from us on the outage. We, again, are very sorry for the disruption this has caused you.

UPDATE

Xero was hit earlier today with an outage that affected many of our customers for approximately two hours.  We’re closely monitoring our systems to ensure everything is healthy again.

We had our dedicated teams working on this issue to get you up and running again and the issue was resolved as quickly as possible. Know that we’re already working to ensure that this doesn’t happen again.

The incident was initially triggered by a hardware outage, which caused instability in our core network.

We don’t have a clear root cause yet. However, the likely reason is that it was a software issue (triggered by the hardware failure) in the switch that corrupted both the primary and redundant network infrastructure.

The way that redundancy works in these situations is that the redundant switches monitor each other and the network and decides when to fail safe over. The system was fully operational when it was last tested two weeks ago. Both the A and B sides of our network were able to run independently and the automated failover between them worked as expected. That didn’t happen this time.

We’re working with the equipment manufacturer and our provider to determine why both sides were impacted, but it’s important to note that there was no data lost and this is not a security issue.

We are migrating to the public cloud which will help ensure this doesn’t happen in the future. The way we will operate in the public cloud is quite different to our current infrastructure and while it removes the likelihood of similar issues, every environment has an element of outage risk.

We know what broke and we knew how to fix it, but we don’t know why it broke, yet. Our team is currently conducting a thorough post-mortem with the hardware manufacturer and our service provider to ensure that we understand the underlying root cause of the issue and better prepare for potential network outages in the future.

Our last major outage was more than five years ago and despite this morning’s outage, we have maintained an uptime of 99.97% for the last 12 months, and will strive to retain our industry-leading 99.99% uptime into the future.

We apologise for letting you down.

283 comments

Julie East
February 24, 2016 at 12.14 pm

Thank you for keeping us informed through the process – appreciate it.

Katy
February 24, 2016 at 12.14 pm

Check your log in people – I’m back in! Can’t see that Xero has updated yet.

Nic
February 24, 2016 at 12.14 pm

Could be so much worse… could be using MYOB!

Christo
February 24, 2016 at 12.19 pm

You took the words right out of my mouth!

Fin Adnam
February 24, 2016 at 12.15 pm

Thanks for the update, earning no money is not ideal, but the swell is running, so Im off for a surf!

Andrea Silvers Xero
February 24, 2016 at 12.16 pm

We have now resolved the issue. Everyone should be back up and running.

Anne in reply to Andrea Silvers
February 24, 2016 at 12.29 pm

Thank you, much appreciated 🙂

Megan in reply to Andrea Silvers
February 24, 2016 at 12.31 pm

Just wanted to say thanks for the communication throughout the morning. Great product, great service. Thanks very much.

Michelle G in reply to Andrea Silvers
February 24, 2016 at 1.32 pm

Thanks. 🙂

Melissa
February 24, 2016 at 12.17 pm

Xero is up and running!!

Leah
February 24, 2016 at 12.18 pm

Yeah Thank you its working now!!!!!

Tim
February 24, 2016 at 12.18 pm

xero is back now. Thanks for the effort.

Lucas
February 24, 2016 at 12.18 pm

Thanks Xero for giving me the day off so I can watch Australia beat New Zealand in the cricket.

Please continue with you’re excellent updates and hopefully things will be up and running once the cricket is finished.

For everyone, else go take a walk, breath and be thankful for your good health.

Christo
February 24, 2016 at 12.19 pm

Thanks to Xero for keeping us in the loop. Hardware failures happen. Nothing is infallible. Sure it’s inconvenient for all of us but they are doing their best to rectify it ASAP. I think it’s a bloody great product at very little cost to our company.

I think it’s important to maintain perspective at times like this.

Robert
February 24, 2016 at 12.19 pm

Anyone remember when Quickbooks Online was down for 3 days in 2012 because someone tripped over an extension cord? talk about incompetence.

Allan Fefergrad
February 24, 2016 at 12.19 pm

We’re back in business!

PHur
February 24, 2016 at 12.19 pm

It’s working again for me too

Salesh Narayan
February 24, 2016 at 12.20 pm

Thanks for the update, we will await for the fix. Xero is great software.

James Anderson
February 24, 2016 at 12.21 pm

Working now, it was only a tad fraught because it’s payday today and us in admin didn’t want to face a mutiny of staff. Some of them are big and kind of scary.

Good job to the techs!

Renae
February 24, 2016 at 12.22 pm

Surely there should backup plans in place for these instances, it’s not Xero’s business that is down there are 1000’s of business that are down. All loosing money waiting… Not a happy customer.

Fred
February 24, 2016 at 12.23 pm

keep drinking that cloud Koolaid. When it goes down/gets hacked MANY businesses are/will be effected -> not just one… anyone with basic IT experience can see the downsides of cloud computing.

jane
February 24, 2016 at 12.24 pm

YAY thanks so much up and running 🙂

Lyn
February 24, 2016 at 12.25 pm

Thanks for keeping us up to date and getting us back up and running. We’ve been moving clients to Xero since 2012 and this is the first time we’ve been unable to access the data at any time anywhere in the world so this is nothing compared to the old desktop days. Keep up the excellent work!

Sarah Bate
February 24, 2016 at 12.28 pm

Thank you for the updates – and now you’re fixed – good job – 2 hours downtime in over 2 years of using Xero? – not bad!

Trisha
February 24, 2016 at 2.26 pm

My second time in two years – of not being able to access.

Tony
February 24, 2016 at 12.31 pm

well done every one, you deserve a drink ?

Cass
February 24, 2016 at 12.33 pm

Have you ever considered a back up plan? Unbelievable that you don’t have one.

Bob
February 25, 2016 at 6.57 am

They detail the backup plan in the blog post.

Chaz
February 25, 2016 at 9.36 am

If you read their update they stated there was a plan but that failed too

James Anderson
February 24, 2016 at 12.34 pm

Oh how quickly some people’s expectations have become totally unreasonable when it comes to technology. Obviously these people weren’t on the internet during dial-up times. If you can do it better, you go invent it. Go on.. shoo.

Kylie Mashford
February 24, 2016 at 2.44 pm

Well said James

Jill
February 24, 2016 at 12.36 pm

Good effort Xero :):)

Anya
February 24, 2016 at 12.38 pm

Thanks for keeping us in the loop and your timely resolution. Can’t imagine we would have such a quick response if our own service was down.

Graham
February 24, 2016 at 12.45 pm

We just came back up. Thanks. Stressful time for us all, but first time in about 5 years, and if I’d been using an installed system it could even more likely be my own system that went down.
Just don’t let it happen again.
Can’t even watch cricket, now.

Carl
February 24, 2016 at 12.52 pm

Thanks Xero, no casualties and life goes on

Kim
February 24, 2016 at 12.52 pm

In the years that I’ve been working with Xero this is the first outage that has ever affected me. These things happen. However, you guys quickly replied to my email, kept me updated and got things running reasonably quickly. The communication was brilliant. Well done Xero!

cheryl
February 24, 2016 at 12.56 pm

thanks – have a nice day

Ian
February 24, 2016 at 1.16 pm

Great work from your team Xero.

Unfortunately you can never 100% rule out outages – it still happens now and then for sites like Facebook, Twitter and YouTube. Unfortunately the proverbial happens, and while we can try and limit it, we can never prevent it completely.

The real measure of a quality software provider is in their response to such an outage. Xero has given us pretty good communication here, with the longest spell between updates being 23 minutes. On top of that, the issue was resolved relatively quickly (I’ve had other outages from other IT providers much longer than that), and most importantly of all, no data was lost.

Obviously a hassle for those who needed to do something right that minute, but definitely reassuring that these outages are rare, and well handled. Thanks team 🙂

Jo
February 24, 2016 at 1.40 pm

You can spot all the people on here who leave all their work till the last minute. 🙂

Wal
February 24, 2016 at 1.43 pm

I have been using Xero for some time now and this is the first time I can recall the system going down. thank you for getting things up and running as quickly as you did and also for keeping us informed through the process.

Andy Jones
February 24, 2016 at 1.51 pm

Well done guys – great transparency and a quick resolution. Keep up the excellent work.

Sonya Henry
February 24, 2016 at 2.14 pm

I can’t log in…

shane
February 24, 2016 at 2.47 pm

let us all not forget that it was not so long ago that all the transactions had to be hand posted into a ledger book, using a fountain pen! From one who has done such work I assure you, nothing is infallible as what can go wrong will go wrong , Murphy’s Law.

Warren
February 24, 2016 at 3.59 pm

Thanks for the updates, and happy to see service is now restored. After reading your update, I am confused with paragraph 7:

“We are migrating to the public cloud which will help ensure this doesn’t happen in the future. The way we will operate in the public cloud is quite different to our current infrastructure and while it removes the likelihood of similar issues, every environment has an element of outage risk.”

Can you please clarify the migration to the public cloud?

Pano
February 24, 2016 at 4.29 pm

You are the best!

I forgive you 🙂

xoxox

Alison
February 24, 2016 at 4.56 pm

Thanks for the information and regular updates. It’s nice to be kept in the loop. It is a pity other large companies don’t respond the same when they have issues. Telco’s in particular.

Can we get clarification on ‘moving to the cloud’ statement.

Thanks

Kazy C
February 24, 2016 at 9.15 pm

Wow, talk about a thorough explanation…awesome…. just like your Rugby players! True professionals that’s what we like.

Steve
February 24, 2016 at 10.58 pm

Lucky is wasn’t myob
They would have blamed the user

Well done people

I love Xero

Tracy
February 25, 2016 at 12.15 am

So impressed with how you’ve kept us informed with your honesty. I’ve spent longer in phone queues to MYOB to hear lies!

You’re still beautiful software to me! 😉

Andrea Bell
February 25, 2016 at 8.40 am

Well done on your communication and keeping everybody informed along the way – made a huge difference. Every company will have times like this but its all about how you handle it. Good job – still love xero!

Lynelle
February 25, 2016 at 10.27 am

Thanks for managing to fix this problem so quickly – pretty amazing – & replying to our queries about it as well. Technology will never be perfect!

Peter
February 25, 2016 at 1.11 pm

I am trying to complete the surveys from the Roadshow in Melbourne and the links provided result in an error message. Second day in a row with problems….oops!

Saheel
February 26, 2016 at 11.08 am

Has anyone else noticed bank statement not being uploaded in Xero this morning. We have three entities and none of the bank statements are uploaded. Banks we use are CBA and Westpac.

Melanie
February 26, 2016 at 12.53 pm

It’s been terrible all morning today. Down, up down, up. What is going on???

Nick Stanley
March 2, 2016 at 5.18 am

This is now getting to be a serious issue; not updated feeds since 23 Feb; causing work backlog and month end delays.Moving into Day 8

Leave a reply

Your email address will not be published. Required fields are marked *