Unexpected Outage (resolved)

We are currently offline due to an issue within our application hosting infrastructure. We have identified the cause of the issue and are working to resolve this.

We will provide any updates, including expected time for the service to be restored, as soon as we have more information.

UPDATE  (Wed 2nd Nov, 12:24pm GMT) – We have the service back online as of about about 15 minutes ago. Continuing to monitor everything closely, but not expecting any further issues. Will provide further information once our internal debrief is complete.

UPDATE  (Thurs 3rd Nov, 3:24am GMT) – A quick update with some further details on this outage. The initial cause was an issue within our application database server cluster, which triggered an automatic failover to the back-up database node. Due to a secondary issue this failover was unable to complete successfully, causing the outage. Resolution involved some storage disk configuration changes, which made up the majority of the time involved.  We are still completing our post incident review, but we have resolved the secondary issue with the automated database failover.

We apologise for the inconvenience caused to any customers by this outage. We continue to make a very significant investment in our hosting platform and resulting service availability level, which we have maintained at 99.99% since we launched Xero in 2007. It frustrates us if we have any unscheduled downtime, but we work hard to learn whatever we can from these incidents and use this to further improve and strengthen our platform going forward.

9 Comments

Sarah Bradley
November 11, 2011 at 12:38 am

As far as the timing of outages go – this one rocks! It coincided with Elevenses for me 😉

It is back up now – well done and THANK YOU!

Jo Wouters
November 11, 2011 at 12:58 am

OK – Xero is okay for us now (13:20 CET – Brussels Time)

Barry
November 11, 2011 at 1:15 am

Does Alastair have a time machine – predicting what will happen tomorrow!

Barry
November 11, 2011 at 1:16 am

I see it is your automatic dating that is adrift

Michelle
November 11, 2011 at 1:30 am

Still having problems here….12:54 GMT

Potential User
November 11, 2011 at 1:58 am

How are we posting these comments in the future? Even the article above was resolved before it was posted! :-)

Howard Cheetham
November 11, 2011 at 5:54 am

Thank you for your speedy response in solving the outage this AM.

Alastair Grigg
November 11, 2011 at 9:01 am

@Barry @Potential User – I don’t, but time machine would be really handy especially in situations like this! The blog post is headlined with date/time of Xero’s Head Office, which being in Wellington, NZ, is currently GMT +13hrs. Hence the post appeared to be a day ahead for many of you. Conscious of this I gave the post update with GMT time noted.

Scott
November 11, 2011 at 8:56 pm

A positive spin here; what internal network boasts a 99.9% uptime,….the benefits of the cloud…

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

7 business apps to take to the beach

Summer has finally arrived in the UK and Brits all over the place are no doubt collectively yearning for a beach holiday. But for small business owners, that heady feeling of freedom is often replaced by worry at the prospect of leaving their business at such a critical time. A OnePoll survey conducted on behalf ...

Payroll for Maryland and Tennessee

Following on from the release of Colorado and Indiana, we’re pleased to announce the availability of Xero Payroll to businesses located in Maryland and Tennessee. Small businesses represent over 97.5 percent of all employers in Maryland and Tennessee. By bringing seamless accounting and payroll to these small businesses we’re helping tackle the complexity of payroll. ...