Yesterday, many of you will be aware we had performance issues that impacted customers and partners, making it hard to use, and for some Xero was not usable for a period of time. Firstly, we are extremely sorry for the disturbance this may have caused you. Secondly, we want to assure you your data was not at risk.
We’ve been analysing the situation overnight and this morning so we could share the root cause and provide an update to you.
On Sunday (NZT), we upgraded the database server that our authentication systems and session management run on to create extra capacity for our growing user base and with the intention of improving performance. This was not considered a high risk technical update, as we have similar server types already running in our production environment. However, this particular database server setup and this database’s particular workload, caused unforeseen performance issues.
Yesterday, as a large part of our customer base came onto the platform, we observed poor performance of that database impacting customer experience. Our team worked as fast as they could to understand the root cause of the issue and engaged our vendors to understand the potential resolution options. After assessing options to either proceed with trying to tune the new servers, or roll back to our previous server configuration from the day prior, we made a decision to roll back as we believed this would give our customers and partners a faster return to normal performance.
We will continue to investigate the dynamics of the technical issues and will provide a more technical update as we resolve all the details. We’re concentrating our investigations on how the server class manages its Central Processing Units (CPUs) and how they access memory.
Please be assured that customer access to our platform and a high performance level is of the utmost importance and we are focussed on delivering a quality experience to you and your business at all times.
With regards to your data and Xero’s core platform, we want to also reassure you that your data was never at any risk, the issue was specifically around server configuration, impacting performance. Xero customer data is stored securely and backed up on Amazon Web Services.
So, thank you for your patience yesterday, we realise this impacted many of our customers. We’re continuing to monitor this issue as a matter of priority.
10:20pm NZT (9:20 GMT)
This issue has now been resolved.
5:55pm NZT (4:55 GMT)
At Xero we’re committed to improving the performance of our product and platform for our customers and partners. Yesterday, as part of this continual improvement work, we upgraded a key database, in what we had hoped, would improve performance for our customers.
As a result of these improvements, we’re unfortunately experiencing performance degradation of some of our services.
We’re sorry this has impacted our users. Please be assured, our team is working on what is expected to be the issue fix, and while we can’t give an exact time for this, we are hoping it won’t be too much longer.
We will keep you updated via our Status Page https://status.xero.com/