Brought to you by

We are improving our infrastructure platform

Posted 3 years ago in Xero news by Duncan Ritchie
Posted by Duncan Ritchie

In January 2013 Xero moved our customers and systems to a new infrastructure that was designed to deliver an improved experience and to allow us to scale up to a million customers.

That platform has served us and our customers well. However, as we continue to grow our customer base, we are mindful of the need to scale our platform to provide customers with a fast, secure and easily accessible cloud platform they can access anytime, from any device connected to the internet. With that in mind, we are shifting the core cloud platform to Amazon Web Services (AWS), providing us with significant potential to continually improve the underlying technologies that help Xero function.

Our new platform has been built on Amazon Web Services (AWS) across its US data centers, and is designed to allow growth to millions of customers and reduce the chances of downtime.

Building on your #beautiful Xero experience

The new AWS platform provides improved levels of security, availability and performance that will support massive future growth. The new platform will enable us to continuously deploy new features to our customers.

What does the move mean for you?

Between June and the end of 2016, we’ll be moving customer data over to AWS and plan to have all customers on the new platform before the end of the year.

These migrations will happen at the least busy times of our customers, which is early Sunday morning, New Zealand time. We’ve chosen the weekends carefully around our global customers’ key accounting activities so we’ll disrupt customers as little as possible – you most probably won’t even notice.

You’ll receive a message within Xero a few days before an organization is due to be moved, and again 15 minutes prior to it going offline for a short period. Our migration is happening customer by customer so only those customers that are being moved on a particular weekend will be affected.

After the migrations, you’ll be able to continue using Xero as you’ve always done.

Keeping you informed

You’ll be able to read more technical detail about the project in a few weeks time in our next blog.

Supporting information will be available for affected customers, including a list of FAQs.

You can also read the May 2015 announcement from Xero.

36 comments

Michael Shields
May 11, 2016 at 11.53 am

Not unexpected and in line with pursuit of the American Market which is already evident in the American spelling and accents on all of the help files and other text in the Xero programme (program for Americans). I am not particularly impressed and am aware that for the time being there is only one undersea cable connecting New Zealand to America. If that goes phut then the whole thing goes phut. Watch this space.

Duncan Ritchie Xero
May 11, 2016 at 2.53 pm

@Michael

All of the Xero systems you use are currently in the US and will continue to be after this change therefore there is no additional risk nor any alignment to our push into the US.

I’m surprised you are getting American spelling in your help files as we have localised spelling specifically for the US market and Australian and New Zealand customer shouldn’t be getting these. I’ll ask one of the team to look into why you are getting the wrong version.

Mitch in reply to Duncan Ritchie
May 11, 2016 at 4.42 pm

As Amazon Web Services has 11 data centre’s, of which one is in Sydney, will there be a point in time where Australian Xero data will ever be stored on Australian soil?

Duncan Ritchie in reply to Mitch Xero
May 12, 2016 at 10.33 am

@Mitch

We have considered this however we have no firm plans to distribute into regional centres. We will continue to consider this as part of our planning process.

Kym Yeoward in reply to Duncan Ritchie
August 11, 2016 at 8.50 pm

Are there possible Australian legal requirements, for records and ledgers to be kept in Australia ?

glenn in reply to Duncan Ritchie
August 11, 2016 at 2.38 pm

As Am I

Cheers Glenn

Gary Smyth
May 11, 2016 at 7.16 pm

Duncan

Is it possible to copy & store our own data locally on our own computers as a precautionary measure? I’m not worried about the data being stored on AWS, that’s fine. It’s just for peace of mind we would like to also have it locally.

Aidan Donnelly
May 12, 2016 at 2.38 am

Where does your european clients data reside? Data stored in USA a big barrier to sales.

Duncan Ritchie in reply to Aidan Donnelly Xero
May 12, 2016 at 10.55 am

@Aidan

We use third party hosting providers’ US servers to store customer data. We only use top tier hosting providers who are committed to satisfying EU adequacy requirements for EU to US personal data transfers. We have in place with all our hosting providers transfer mechanisms that satisfy the requirements relating to Xero’s transfer of data from the EU to the US. We do not rely on the Safe Harbor framework in light of the Schrems decision and we instead make use of Model Clauses and other contractual protections. Many of our EU based competitors also store data on US servers so please do not assume otherwise.

Jordan Zoot
May 13, 2016 at 6.16 am

With the move to AWS how about deploying SSO under IAM so we have real SSO option?

Duncan Ritchie in reply to Jordan Zoot Xero
May 13, 2016 at 9.55 am

@Jordan

We have planned work to improve our SSO capabilities. I don’t have a date yet sorry so I can’t be specific on the timing.

Mark Lawton
May 31, 2016 at 11.45 am

Duncan will the move allow greater transaction limits for Xero customers? currently I believe there is a soft limit of 1000/month. It would be good to know where the cutoff would be before having to move to another system to plan for customer growth and whether these limits will increase in the future as many companies that started out with Xero years ago are hitting the 1000/month transactions but don’t want to move to another system.

Duncan Ritchie in reply to Mark Lawton Xero
June 1, 2016 at 4.42 pm

@Mark. The move will have some impact on transaction capability and we have a bunch of other work that we are doing to improve performance for larger organisations. It would be good to understand if you have particular pain points. If you send in a support ticket attention to me I’ll pass it to the right people.

alison
June 22, 2016 at 1.03 pm

Will this upgrade address the payroll hanging issue (Once we hit approve, there’s no editing payslips without Xero timing out and returning an error). Hoping so!

T
June 23, 2016 at 12.32 pm

Upgrades have to happen that’s fine but NOT impressed that we may not have access to one or more of our businesses on the weekend before EOFY… Not very good planning..

Dave
July 7, 2016 at 11.39 pm

is this pure infrastructure, or will there be feature updates as well? Scaling to a million + customers who still can’t do basic stuff like submit multi-currency expense sheets is going to be hell on your support crew…

Tony Whale
July 21, 2016 at 8.50 pm

Indeed a grump of mine, down played as a trivial unimportant request.

Duncan Ritchie in reply to Dave Xero
July 24, 2016 at 12.19 pm

This project is just about the platform we run on. It doesn’t have any goals around functional changes to the software.

Most of our teams are still working on functional improvements.

Craig
July 21, 2016 at 1.09 pm

Thought you guys would have embraced Azure. Seems to be a much better choice moving forward than AWS. Shame.

Duncan Ritchie in reply to Craig Xero
July 24, 2016 at 12.21 pm

We looked at a number of options including Azure but concluded that AWS better suited the workloads we have and intend to have. We do have some aspects of our production platform and Internal IT environments on Azure.

Richard McLean
August 11, 2016 at 9.10 pm

Craig – Azure would be a huge step backwards and everyone should be commending Xero for the move AWS…

Great work guys.

Tony
August 12, 2016 at 1.04 am

I feel much better it’s moving to AWS. They are the clear leaders in providing secure cloud infrastructure which is important for my customer data. Take a look at the Gartner Magic quadrant.

Sara Robeson
July 23, 2016 at 4.04 am

How long can we expect service to be down on 7/23?

Duncan Ritchie in reply to Sara Robeson Xero
July 24, 2016 at 12.21 pm

We aim for < 4 hours of downtime.

jan Flanagan
July 24, 2016 at 10.24 am

Thanks heaps Xero – a few days before BAS due and you are offline!!! Not good timing.

Duncan Ritchie in reply to jan Flanagan Xero
July 24, 2016 at 12.23 pm

Sorry Jan I know this migration impacts customers and that can be frustrating. We try to minimise the impact by performing the work early in the day for Australia so that the bulk of our customers won’t be impacted.

Allan
August 3, 2016 at 7.57 pm

hi,
is it possible to have an email send to us when the work is complete this time and reassurance that the transfer is fully tested before opening the system?
Also an ETA of completion would be useful for our business planning.

Last time the AWR transfer work occurred , we signed on and found the balances to be wrong – which to be fair was eventually fixed but did shake our confidence in your company. We are hoping there isn’t a repeat ..

thanks
Allan (UK )

Mark Rees
September 20, 2016 at 1.59 pm

Allan,

We do understand that these migrations can have a big impact on some customers and I can reassure you that the transfer is thoroughly tested before, during and after the migration.

The best source of information about the outage period is found here: https://www.xero.com/blog/offline-status-page/. In the rare cases that we need more time, we will notify all impacted customers.

Mark Rees
GM, Platform Architecture and Delivery
Xero

Michael
August 11, 2016 at 9.25 pm

Been using Xero since July 2015, most reliable service Ive experienced. Advanced notice of being offline for a few hours is fine with me, throw in its a weekend even better, throw in that time slot – I will never even notice. Thanks to the Xero team doing the work during those unsociable hours so Im not impacted, I do appreciate the effort.

Accounts
August 12, 2016 at 7.18 am

Hi Xero,
Can you provide details of data privacy given the propensity of the NSA to scrape all data stored on US servers? I would be interested in how NZ and EU data privacy legislation is being met.
Thanks,
Accounts

Mark Rees
September 20, 2016 at 2.06 pm

Hi,

For detailed information about our approach to privacy please refer to this page: https://www.xero.com/nz/about/privacy/

Mark Rees
GM, Platform Architecture and Delivery
Xero

Andrew
August 21, 2016 at 9.26 pm

Good news to see AWS come into the picture, but most countries have issues around data sovereignty, Xero would be out of business if they did not comply with regulatory requirements. The only issue I have is how unreliable their bank interfaces are. They quite often miss transactions and you end up with balances that do not match after a reconciliation. It’s becoming a painful product.

Jay
August 22, 2016 at 1.39 am

As you move to the public cloud I need to know of data privacy in much more detail. Can you please publish relevant documents on your website and make sure people have an easy way out if you don’t meet appropriate requirements. Also, please make sure you address the issue of the UK leaving EU appropriately.

On the other hand – your API doesn’t work most of the weekend which makes all most of the processes broken. Your website (https://www.xero.com/blog/offline-status-page/?mkt_tok=eyJpIjoiTVdOa1l6VTRZelJtTURVMyIsInQiOiJic1N2ZVQ1dGtmZExKS1BTb243b093OURFM2pZTWpjd29IWjBoaDd4Rmg0bHYxR1wvV01QVE9OWVlBdjlFYlVtZms0WERtd051TWljaGRnNjVTa1J0Zk1cL3J6UGx5SG1uYjNTeXBLWHk5cmZZPSJ9) mentions that you have an interim solution in place:

“Update as at 6.25pm New Zealand time, Sunday 21 August:
We have an interim solution in place for the API issue we’ve been experiencing. Xero’s API is now operational, however please be aware that our API Partners using our ecosystem applications, and our Practice applications: Xero Practice Manager, WorkflowMax, Workpapers, and Tax may experience some degraded performance or slowness. Our teams are continuing to work on a final resolution that sees our API and Practice tools up and running at full expected performance levels. Thanks again for your patience while we continue to resolve these issues. We are very sorry for any inconvenience this has caused.”

but it’s clearly not the case. Also, the last update is from 7 hours ago and we are left in the dark as what’s going on with API problem which you seem you haven’t anticipated and which is very worrying indeed.

Mark Rees
September 20, 2016 at 2.05 pm

Jay,

The best place to look for information about our privacy policy is here: https://www.xero.com/nz/about/privacy/

I hope you are no longer experiencing issues due to the migration on the 21st of August. If you are having problems related to this or any other issue, please make sure you contact Xero support.

Regards,
Mark
GM, Platform Architecture and Delivery

Jay
March 1, 2017 at 8.16 am

It’s a second major outage on AWS I saw (I don’t know how many more they were, hopefully someone at Xero is counting them and making some stats). I know AWS cloud is solid and it’s difficult to have something more robust and flexible at the same time in terms of the platform but how well you are prepared for the mother of all outages when your service gets staffed not for a few hours but a few days or a week? Are you able to move your infrastructure anywhere else in case AWS cloud will fail you big time?

I’m only asking because I’m sitting here in front of my computer seeing how you are completely reliant on AWS and dependent on their engineers without any control whatsoever to do anything about your business if things go bad. And it does worry me because there is my business data within this stack.

So my question is – if you don’t have a contingency plan (which is probably the most likely scenario – relying on AWS redundancy instead of building your own was probably one of the reasons you moved) what should I do if by a very minimal chance AWS goes burst but I still need my data? Should I export it on daily basis? Can you provide us with automatic backup? Are you backing up our data yourself to a provider independent of AWS? What are your RTOs and RPOs in an event of major outage?

Leave a reply

Your email address will not be published. Required fields are marked *