Unexpected Outage (Resolved)

We have had to take Xero offline due to an issue within the application hosting infrastructure.

We have identified the issue and our team is working on resolving it. We’ll provide an update shortly on when we expect to have Xero back online.

Our apologies for any inconvenience this outage is causing.

(UPDATE) We are still working on fixing the issue, but expect to be back online within the next 30 minutes.

(UPDATE) We have encountered some further delays in bringing the system back online, which we expect will take around another 30 minutes to resolve. At this stage we now expect to be back online by 8:45pm (GMT) or 8:45am (NZT).

(UPDATE) The system is back up now and we have the team closely monitoring the system.  Apologies for the inconvenience this outage caused.

(UPDATE) At 6:00am NZT (7:00pm GMT) one of our database servers failed and cut over automatically to its standby server.  The standby server failed to start due to issues with one of the database files.  The Xero operations team and Rackspace, our hosting provider, were forced to recreate the damaged file in order to recover the database server.  There was no loss of data as a result of this issue however the application was offline for approximately two and one half hours.  We have a number of levels of redundancy within our hosting environment that typically means our system is not impacted by infrastructure issues.  Unfortunately in this situation there was no way for the automated recovery to resolve the issue.  We are currently planning an upgrade to the latest version of the database platform that supports more advanced resilience, which should further reduce the risk of a similar outage.

7 Comments

Matt Hixson
July 9, 2012 at 7:51 am

I appreciate the communication.

Mike Block CPA
July 9, 2012 at 7:59 am

When I try to login, a screen say this is a scheduled maintenance outage. It is not. There also is no time in the post (or the scheduled maintenance screen) showing when the outage started and when you feel it may end. There also is no separate RSS feed for such outages, as there should be. Finally, there should be a page showing the history of scheduled and unscheduled outages.

Gayle Buchanan
July 9, 2012 at 8:05 am

All was well at 5.30am – 6.30am NZ time …. then thump – I’m sure all will be up and running – in the meantime …. yes I’m sad … just love no noise in the house that early

Nigel McNie
July 9, 2012 at 8:28 am

Very cool outage page, +5 points to xero 😉

Also… monday morning outages, waaa. I feel for you.

Guy Pearson
July 9, 2012 at 9:32 am

@ali

Thanks for the communication on the issue.

Like many, (many moons ago) i worked for a firm with a server that would create downtime every other day for ~ 1/2 hour.

I’m sure we can all tolerate these little hitches.

Coffee time anyone?

Cheers.
Guy

Elizabeth Salter
July 9, 2012 at 1:03 pm

Thanks for keeping us updated. It’s much appreciated. :)

Alex Haines
July 9, 2012 at 7:05 pm

The most important thing is that during this down-time you communicated with us. So many companies DO NOT do this and it is soooo easy!

Thanks!

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

Introducing more flexibility for UK Payroll

Payroll in Xero is flying in the UK. Since its launch earlier this year, our customers in the UK have processed nearly £200 million of employee pay using it. Our Payroll team has been focused on extending and refining the core experience of using Payroll in Xero, guided by feedback from the thousands of businesses, ...

Automatic enrolment and pension schemes in Xero

Automatic enrolment of employees into a workplace pension will affect every employer over the next couple of years. Every employer has a legal duty to enrol all eligible employees into a pension scheme. They also need to assess their employees according to certain criteria to find out who is  ‘Eligible’. We’re building assessment, auto enrolment, and workplace pension capabilities ...