Brought to you by

[RESOLVED] Xero is experiencing issues

Posted 4 years ago in Xero news by xero
Posted by xero

Xero is currently experiencing an issue with the Xero websites.  The Operations and Infrastructure teams are investigating this as our highest priority to resolve the issue. We will continue to update this blog with more developments until performance is returned to normal. We’ll update as soon as we have any details.

We acknowledge the frustration this may be causing, and thank you for your support while we pull out all the stops to get this fixed.

UPDATE – RESOLVED 10:40PM GMTWe are back live. The Xero Operations team is currently investigating root causes, continuing to monitor the situation.


On 15 February at 9:53 PM GMT we experienced a 40 minute outage to our services.

The root cause analysis has identified a bug in our management software that impacted a change to our infrastructure load balancers.

We were unable to back out the change as quickly as hoped and apologise to customers for the inconvenience and interruption to service.



Catherine Xero
February 15, 2014 at 11.33 am

Hi folks – the outage is affecting WorkflowMax to.

February 15, 2014 at 11.39 am

Any ideas on timing? – Thanks

February 15, 2014 at 11.40 am

Looks like its working again! Thanks Guys!

Diane Crawford-Errington
February 15, 2014 at 11.43 am

And Workpapers – seems it’s across the board. Thanks for the update team.

Anton Gerner
February 15, 2014 at 3.17 pm

Any updates about what actually happened?

Heather Smith
February 15, 2014 at 11.33 pm

Are we blaming cupid on Valentines Day?

Jason Danner
February 15, 2014 at 11.54 pm

The Careers page still seems to be down. Everything else is up, though.

Anton Gerner
February 16, 2014 at 1.19 pm

Can you please explain in English: “The root cause analysis has identified a bug in our management software that impacted a change to our infrastructure load balancers”

What are “load balancers”?

David Wills in reply to Anton Gerner
February 17, 2014 at 10.57 am

@Anton – Hi Anton – yes sorry now in plain English; we had a technical problem on our network on Saturday and had to revert a change that didn’t work as expected. A ‘load balancer’ spreads user requests over a fleet of computers to distribute and scale our user access. We’ll work on improving our communications to keep you better informed of service issues in the future.

mike silich
February 16, 2014 at 5.11 pm

Hi Guys, Great that the problem was fixed reasonably quickly – bit scarey when our businesses are so dependent on xero.
My disappointment was that the blog notification that you were experiencing was only posted about the time you came live. The support email link in the help section also wouldn’t send – so we all where in the dark as to what was going on.
System status needs to be much much more prominent.

Mike – Melbourne

Robbie Dellow
February 17, 2014 at 1.30 am

So to rectify the situation you rolled back the change.
What procedures do you have in place to ensure things like this do not happen? Do you not roll out firstly to a dev environment?. Notwithstanding this, I would have thought that any affect on server load balancing would be picked up very soon after the change was implemented and thus the roll back occur before many users get impacted.
Incidentally, not knowing the setup of Xero’s servers, did this performance impact affect all geographical locations?

David Wills in reply to Robbie Dellow
February 17, 2014 at 11.03 am

@Robbie – Hi Robbie – yes that’s correct, we rolled back the change that caused the outage for our users. The change had been tested prior to implementation. However, it didn’t identify the issue we encountered on Saturday and therefore our investigations took longer to fix and restore the normal service than we hoped. The outage was experienced in all geographic regions and if interested, we explain more about the Xero platform on our blog: Apologies for any disruption that this caused you and our Xero community.

Robbie Dellow
February 18, 2014 at 6.42 pm

Thanks for replying, David.
I would have thought a software change/update would be propogated incrementally to each geographical location rather than exposing yourself to the risk of ‘blowing away’ Xero globally, which unfortunately half-pie happened in this instance.
From my global Unix sys admin/market data experience in merchant banking, any rollout of changes, would, at the most, affect a region. To do otherwise would endanger a banks reputation and livelihood.
From what I understand of Xero architecture and software update rollouts, you do propogate certain geographic-specific s/ware updates which I assume would thus only go to those servers that service that geographic location that the update is intended for.
Excited for the huge potential of Xero, but the required rapid evolution of the software means more and more rollouts. Sure hope Xero can preserve the 99% uptime figure it claims. Don’t want to give the incumbents any fodder.
I see this incident a bit more worrying than seems implied ( or compared to other incidents), in such that we are talking global. With Xero’s increasingly broad global base this will surely be paramount to avoid.

Leave a reply

Your email address will not be published. Required fields are marked *