Unexpected Outage (resolved)

We are currently offline due to an issue within our application hosting infrastructure. We have identified the cause of the issue and are working to resolve this.

We will provide any updates, including expected time for the service to be restored, as soon as we have more information.

UPDATE  (Wed 2nd Nov, 12:24pm GMT) – We have the service back online as of about about 15 minutes ago. Continuing to monitor everything closely, but not expecting any further issues. Will provide further information once our internal debrief is complete.

UPDATE  (Thurs 3rd Nov, 3:24am GMT) – A quick update with some further details on this outage. The initial cause was an issue within our application database server cluster, which triggered an automatic failover to the back-up database node. Due to a secondary issue this failover was unable to complete successfully, causing the outage. Resolution involved some storage disk configuration changes, which made up the majority of the time involved.  We are still completing our post incident review, but we have resolved the secondary issue with the automated database failover.

We apologise for the inconvenience caused to any customers by this outage. We continue to make a very significant investment in our hosting platform and resulting service availability level, which we have maintained at 99.99% since we launched Xero in 2007. It frustrates us if we have any unscheduled downtime, but we work hard to learn whatever we can from these incidents and use this to further improve and strengthen our platform going forward.

9 Comments

Sarah Bradley
November 3, 2011 at 12:38 am

As far as the timing of outages go – this one rocks! It coincided with Elevenses for me 😉

It is back up now – well done and THANK YOU!

Jo Wouters
November 3, 2011 at 12:58 am

OK – Xero is okay for us now (13:20 CET – Brussels Time)

Barry
November 3, 2011 at 1:15 am

Does Alastair have a time machine – predicting what will happen tomorrow!

Alastair Grigg in reply to Barry
November 3, 2011 at 9:01 am

@Barry @Potential User – I don’t, but time machine would be really handy especially in situations like this! The blog post is headlined with date/time of Xero’s Head Office, which being in Wellington, NZ, is currently GMT +13hrs. Hence the post appeared to be a day ahead for many of you. Conscious of this I gave the post update with GMT time noted.

Barry
November 3, 2011 at 1:16 am

I see it is your automatic dating that is adrift

Michelle
November 3, 2011 at 1:30 am

Still having problems here….12:54 GMT

Potential User
November 3, 2011 at 1:58 am

How are we posting these comments in the future? Even the article above was resolved before it was posted! 🙂

Howard Cheetham
November 3, 2011 at 5:54 am

Thank you for your speedy response in solving the outage this AM.

Scott
November 3, 2011 at 8:56 pm

A positive spin here; what internal network boasts a 99.9% uptime,….the benefits of the cloud…

Leave a Reply

Your email address will not be published. Required fields are marked *

South Africa + Xero = Love

Although South Africa has had its fair share of economic challenges, the country’s entrepreneurial spirit is incredible. We’re seeing a major shift to the cloud as many small businesses embrace new technology. This was a central theme of last week’s Xero Roadshow events as they hit Johannesburg and the beautiful Cape Town. Gary Turner and ...

Our newest release lets you create more from a copied invoice

We know that every day, thousands of you copy your invoices instead of creating new ones. This saves time and reduces one of the most repetitive tasks many of you do. With this in mind, we’ve decided to extend the copy functionality to allow you to create new Quotes, Bills and Purchase Orders using the details ...