Unexpected Outage (resolved)

We are currently offline due to an issue within our application hosting infrastructure. We have identified the cause of the issue and are working to resolve this.

We will provide any updates, including expected time for the service to be restored, as soon as we have more information.

UPDATE  (Wed 2nd Nov, 12:24pm GMT) – We have the service back online as of about about 15 minutes ago. Continuing to monitor everything closely, but not expecting any further issues. Will provide further information once our internal debrief is complete.

UPDATE  (Thurs 3rd Nov, 3:24am GMT) – A quick update with some further details on this outage. The initial cause was an issue within our application database server cluster, which triggered an automatic failover to the back-up database node. Due to a secondary issue this failover was unable to complete successfully, causing the outage. Resolution involved some storage disk configuration changes, which made up the majority of the time involved.  We are still completing our post incident review, but we have resolved the secondary issue with the automated database failover.

We apologise for the inconvenience caused to any customers by this outage. We continue to make a very significant investment in our hosting platform and resulting service availability level, which we have maintained at 99.99% since we launched Xero in 2007. It frustrates us if we have any unscheduled downtime, but we work hard to learn whatever we can from these incidents and use this to further improve and strengthen our platform going forward.

9 Comments

Sarah Bradley
November 3, 2011 at 12:38 am

As far as the timing of outages go – this one rocks! It coincided with Elevenses for me 😉

It is back up now – well done and THANK YOU!

Jo Wouters
November 3, 2011 at 12:58 am

OK – Xero is okay for us now (13:20 CET – Brussels Time)

Barry
November 3, 2011 at 1:15 am

Does Alastair have a time machine – predicting what will happen tomorrow!

Barry
November 3, 2011 at 1:16 am

I see it is your automatic dating that is adrift

Michelle
November 3, 2011 at 1:30 am

Still having problems here….12:54 GMT

Potential User
November 3, 2011 at 1:58 am

How are we posting these comments in the future? Even the article above was resolved before it was posted! :-)

Howard Cheetham
November 3, 2011 at 5:54 am

Thank you for your speedy response in solving the outage this AM.

Alastair Grigg
November 3, 2011 at 9:01 am

@Barry @Potential User – I don’t, but time machine would be really handy especially in situations like this! The blog post is headlined with date/time of Xero’s Head Office, which being in Wellington, NZ, is currently GMT +13hrs. Hence the post appeared to be a day ahead for many of you. Conscious of this I gave the post update with GMT time noted.

Scott
November 3, 2011 at 8:56 pm

A positive spin here; what internal network boasts a 99.9% uptime,….the benefits of the cloud…

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

Payroll for South Dakota, Wyoming, Washington

Following up on the continued momentum from the last release of New Hampshire — we’re excited to announce Payroll in Xero availability for South Dakota, Wyoming and Washington. Over 100,000 small businesses in South Dakota, Wyoming and Washington employ at least one employee. Xero is truly excited to provide an accounting and payroll solution that ...

The new look Xero Android app

We’re pleased to announce the biggest release since the native version of our Xero Android app. Every pixel of the dated Android 4.x ‘Holo’ look is gone! Xero now follows Google’s ‘Material’ design guidelines. We’ve revisited every single screen with an eye on consistency and #beautiful design. We think you’ll find it a vast improvement. ...