Unexpected Outage (resolved)

We are currently offline due to an issue within our application hosting infrastructure. We have identified the cause of the issue and are working to resolve this.

We will provide any updates, including expected time for the service to be restored, as soon as we have more information.

UPDATE  (Wed 2nd Nov, 12:24pm GMT) – We have the service back online as of about about 15 minutes ago. Continuing to monitor everything closely, but not expecting any further issues. Will provide further information once our internal debrief is complete.

UPDATE  (Thurs 3rd Nov, 3:24am GMT) – A quick update with some further details on this outage. The initial cause was an issue within our application database server cluster, which triggered an automatic failover to the back-up database node. Due to a secondary issue this failover was unable to complete successfully, causing the outage. Resolution involved some storage disk configuration changes, which made up the majority of the time involved.  We are still completing our post incident review, but we have resolved the secondary issue with the automated database failover.

We apologise for the inconvenience caused to any customers by this outage. We continue to make a very significant investment in our hosting platform and resulting service availability level, which we have maintained at 99.99% since we launched Xero in 2007. It frustrates us if we have any unscheduled downtime, but we work hard to learn whatever we can from these incidents and use this to further improve and strengthen our platform going forward.

9 Comments

Sarah Bradley
November 3, 2011 at 12:38 am

As far as the timing of outages go – this one rocks! It coincided with Elevenses for me 😉

It is back up now – well done and THANK YOU!

Jo Wouters
November 3, 2011 at 12:58 am

OK – Xero is okay for us now (13:20 CET – Brussels Time)

Barry
November 3, 2011 at 1:15 am

Does Alastair have a time machine – predicting what will happen tomorrow!

Barry
November 3, 2011 at 1:16 am

I see it is your automatic dating that is adrift

Michelle
November 3, 2011 at 1:30 am

Still having problems here….12:54 GMT

Potential User
November 3, 2011 at 1:58 am

How are we posting these comments in the future? Even the article above was resolved before it was posted! :-)

Howard Cheetham
November 3, 2011 at 5:54 am

Thank you for your speedy response in solving the outage this AM.

Alastair Grigg
November 3, 2011 at 9:01 am

@Barry @Potential User – I don’t, but time machine would be really handy especially in situations like this! The blog post is headlined with date/time of Xero’s Head Office, which being in Wellington, NZ, is currently GMT +13hrs. Hence the post appeared to be a day ahead for many of you. Conscious of this I gave the post update with GMT time noted.

Scott
November 3, 2011 at 8:56 pm

A positive spin here; what internal network boasts a 99.9% uptime,….the benefits of the cloud…

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

Xero staff and partners star in the Xero Band

By Kevin Wood – Direct Sales Manager It’s been two weeks since Xerocon Melbourne and we’re loving all the great feedback that’s come through. It’s fantastic to hear what everyone loved the most. Thanks so much to everyone who attended and helped make it happen! The Xero Band rocks for the fourth year running As ...

New IDC report shows Xero is a global growth leader

Xero is the fastest growing financial applications vendor in the world according to the International Data Corporation’s new report. IDC, a world leader in technology research, recently released their annual report. They report on market sizing and share for finance and financial payroll software providers*. Not only was Xero the fastest growing financial application, it ...