Brought to you by

We’re sorry – really sorry…

Posted 2 weeks ago in Advisors by Tony Stewart
Posted by Tony Stewart

Yesterday, Wednesday 12 September, we experienced a technical outage which affected our users’ access to Xero.

The site went down at approximately 5pm New Zealand time, and was restored at 12.30am New Zealand time.

We know you’re wondering what happened. The nature of the problem made it very hard to diagnose. There are many problems that can exhibit the same symptoms.

We discovered a change was made to our network configuration. The change was incorrectly applied which meant some requests to servers inside that network failed. Our team worked around the clock to identify and implement a solution and to get the site up and running as quickly as possible.

Once the change was detected and reversed, restoration of service was instant, so customers could immediately access the Xero site again. We sincerely apologise for any impact this had on our customers’ working day.

We’re doing everything we can to build more resilience into our platform and reduce any risk of a similar issue in the future. Please rest assured that your data was not at risk at any stage.

This outage was disruptive, and we’re really sorry for the frustration and difficulty it caused you and your business.

We appreciate you bearing with us yesterday, and we’re disappointed we’ve let you down. Our site performance is extremely important to everyone at Xero, and we’re never happy with an outage of any length.

Tony Stewart (Chief Product, Platform and Data Officer), and the Xero Team.

14 comments

Mike
September 13, 2018 at 9.26 pm

I’m pleased you publicly announced what the cause was. However, who made that change and why?

If it was on a critical system, why was it done out of hours without warning and why did you have to “discover it”?

it makes no sense.

Joanne Tait in reply to Mike Xero
September 18, 2018 at 3.20 pm

Hi Mike,

Thanks for your questions. Due to their technical nature, it’s best if we speak directly. Either Tony or one of his direct reports will be in touch with you in the next day or two to go over your concerns.

Thanks,
Joanne

Shelley Pritchard
September 13, 2018 at 11.31 pm

You handled it brilliantly Xero, with regular updates we knew what was going on.

Yes, it caused distruption but not for too long.

We still love you and all that you given us Xero.

Ben
September 14, 2018 at 10.31 am

You must have received different updates to us? All we got was “we’re working on it”!

Trish Scally
September 13, 2018 at 11.41 pm

The outage was very lengthy and extremely disruptive to our business. What compensation will Xero offer?

Joanne Tait in reply to Trish Scally Xero
September 14, 2018 at 12.24 pm

Hi Trish,

We understand that the outage was disruptive to many businesses, and we’re reviewing requests for compensation on a case-by-case basis. Please email our team at support@xero.com, and we’ll be happy to look into this.

Shane Turner
September 13, 2018 at 11.51 pm

Thanks for the info. I noticed it was down so went to the beach instead – easy!

Brett Rowe
September 14, 2018 at 12.15 am

Hi Tony,

Tough day for you yesterday,

However, somethings need further explanation.

1. Why was this ‘change’ made in the live environment and not a test environment? The way you make it sound is that someone made a rogue change which is even more worrying!
2. Why did one change take down the whole thing? Are your server farms not segmented to ensure one mistake doesn’t impact everyone?

All in all your RFO gives more questions (and worries) than I had before you sent it, pretty sure that wasn’t your intention.

My company builds clouds for customers, what you discribe seems very much out of line with standard practice in the industry and whilst I appreciate the audience may not all be cloud builders, I am sure they would like to know why this will never happen again and how you can ensure it.

If you have a full technical RFO I’d welcome the chance to see it, our business was heavily impacted yesterday and I’d love to go back to having full and total confidence in Xero as we had before this incident.

Managing applications for blue chip customers as we do, I know you’ll have a level of detail beyond what you have written and I’d like the chance to see it.

Thank you
Brett Rowe,
Managing Director
Securus Communications Ltd

Joanne Tait in reply to Brett Rowe Xero
September 18, 2018 at 3.09 pm

Hi Brett,

Thank you for your comments. Due to the technical nature of your questions, it’s best that we get in touch with you directly to discuss your concerns. Either Tony or one of his direct reports will be in contact with you in the next day or two to talk about this.

Cheers,
Joanne

Terry Blevins
September 14, 2018 at 3.17 am

Thank you for the sincere apology and why it happwn. Much appreciated. A real taste of good customer service.

Alan H Boal
September 15, 2018 at 4.49 am

I have to say that Xero’s uptime is pretty good! Therefore I can accept this occasional outage and just do something else during that time.

Jon Jenkins
September 15, 2018 at 9.44 pm

I had a team of people unable to work on clients accounts and clients unable to access for the best part of 2/3 of a working day. The length of time it took to detect and resolve the problem is not befitting of what we expect from Xero. Simply not good enough. Anyone remember what happened when Blackberry couldn’t sort out emails for a few days.

Scott
September 19, 2018 at 8.45 am

A really simple request. Can you please put dates in your posts. This one starts off with “Yesterday, we experienced a …” and under the title is says “Posted 6 days ago in Advisors”. Just put a date in the post “Yesterday (Wednesday 13 Sep 2018), we experienced a …” We should have to figure out when a event like this happened.

Joanne Tait in reply to Scott Xero
September 19, 2018 at 12.14 pm

Hi Scott, thanks for your feedback. I’ll update the post with the date included.

Leave a reply

Your email address will not be published. Required fields are marked *