Skip to content

[RESOLVED] Xero is experiencing issues

Xero is currently experiencing an issue with the Xero websites.  The Operations and Infrastructure teams are investigating this as our highest priority to resolve the issue. We will continue to update this blog with more developments until performance is returned to normal. We’ll update as soon as we have any details.

We acknowledge the frustration this may be causing, and thank you for your support while we pull out all the stops to get this fixed.

UPDATE - RESOLVED 10:40PM GMTWe are back live. The Xero Operations team is currently investigating root causes, continuing to monitor the situation.

UPDATE: 12:04 AM GMT 

On 15 February at 9:53 PM GMT we experienced a 40 minute outage to our services.

The root cause analysis has identified a bug in our management software that impacted a change to our infrastructure load balancers.

We were unable to back out the change as quickly as hoped and apologise to customers for the inconvenience and interruption to service.

 

 

Read more about Company News

 

13 comments

Catherine
15 February 2014 #

Hi folks – the outage is affecting WorkflowMax to.

Paul
15 February 2014 #

Any ideas on timing? – Thanks

Justen
15 February 2014 #

Looks like its working again! Thanks Guys!

Diane Crawford-Errington
15 February 2014 #

And Workpapers – seems it’s across the board. Thanks for the update team.

Anton Gerner
15 February 2014 #

Any updates about what actually happened?

Heather Smith
15 February 2014 #

Are we blaming cupid on Valentines Day?

Jason Danner
15 February 2014 #

The Careers page still seems to be down. Everything else is up, though.

Anton Gerner
16 February 2014 #

Can you please explain in English: “The root cause analysis has identified a bug in our management software that impacted a change to our infrastructure load balancers”

What are “load balancers”?

mike silich
16 February 2014 #

Hi Guys, Great that the problem was fixed reasonably quickly – bit scarey when our businesses are so dependent on xero.
My disappointment was that the blog notification that you were experiencing was only posted about the time you came live. The support email link in the help section also wouldn’t send – so we all where in the dark as to what was going on.
System status needs to be much much more prominent.
thanks

Mike – Melbourne

Robbie Dellow
17 February 2014 #

So to rectify the situation you rolled back the change.
What procedures do you have in place to ensure things like this do not happen? Do you not roll out firstly to a dev environment?. Notwithstanding this, I would have thought that any affect on server load balancing would be picked up very soon after the change was implemented and thus the roll back occur before many users get impacted.
Incidentally, not knowing the setup of Xero’s servers, did this performance impact affect all geographical locations?

David Wills
17 February 2014 #

@Anton – Hi Anton – yes sorry now in plain English; we had a technical problem on our network on Saturday and had to revert a change that didn’t work as expected. A ‘load balancer’ spreads user requests over a fleet of computers to distribute and scale our user access. We’ll work on improving our communications to keep you better informed of service issues in the future.

David Wills
17 February 2014 #

@Robbie – Hi Robbie – yes that’s correct, we rolled back the change that caused the outage for our users. The change had been tested prior to implementation. However, it didn’t identify the issue we encountered on Saturday and therefore our investigations took longer to fix and restore the normal service than we hoped. The outage was experienced in all geographic regions and if interested, we explain more about the Xero platform on our blog: http://www.xero.com/blog/2013/11/xero-platform-whats-hood/. Apologies for any disruption that this caused you and our Xero community.

Robbie Dellow
18 February 2014 #

Thanks for replying, David.
I would have thought a software change/update would be propogated incrementally to each geographical location rather than exposing yourself to the risk of ‘blowing away’ Xero globally, which unfortunately half-pie happened in this instance.
From my global Unix sys admin/market data experience in merchant banking, any rollout of changes, would, at the most, affect a region. To do otherwise would endanger a banks reputation and livelihood.
From what I understand of Xero architecture and software update rollouts, you do propogate certain geographic-specific s/ware updates which I assume would thus only go to those servers that service that geographic location that the update is intended for.
Excited for the huge potential of Xero, but the required rapid evolution of the software means more and more rollouts. Sure hope Xero can preserve the 99% uptime figure it claims. Don’t want to give the incumbents any fodder.
I see this incident a bit more worrying than seems implied ( or compared to other incidents), in such that we are talking global. With Xero’s increasingly broad global base this will surely be paramount to avoid.

Add your comment





We welcome all feedback but prefer a real name and email address.