Brought to you by

Xero’s platform structure and history

Posted 1 year ago in Platform by Mark Rees
Posted by Mark Rees

In May, we let you know about the improvements we’re making to our infrastructure platform to ensure it’s ready for the next phase of our growth. We wanted to share why we’re making these changes and how the new platform will work, starting with a bit of history about the way Xero’s platform has evolved. By ‘platform,’ we mean all of the servers, network, storage and so on that we use to run Xero’s applications. The changes to Xero’s platform structure discussed in this post are happening at the same time as we migrate to Amazon Web Services.

The early days

In the early days of Xero, we used a single database to store all our customer information from financial data, to billing, bank feeds and user records. This database was easy to work with and we made sure it was resilient, but as we grew it struggled to handle the load generated by lots of new customers.

Xero arcitecture

In recent years

To make sure our platform could handle our growth we changed the way we stored customer data to use ‘sharding’. Sharding is a common method that involves dividing customers up into groups and allocating those groups to their own dedicated database clusters. At Xero, we allocated 40,000 subscriptions to each database cluster.  Whenever we needed to increase the capacity of the platform, we created a new database cluster and added new customers to it as they signed up. When a database cluster  held 40,000 subscriptions,  we provisioned a new cluster and repeated the process.

Sharding allowed us to smoothly expand the platform as we grew from 10,000 to 700,000 subscriptions. Although that scaling strategy was effective to get to 700,000 subscriptions, all scaling strategies have their limits, and a change in approach is needed to deal with the next phase of growth. This is the point we’ve reached at Xero –  the database scaling approach we have used for the last three years is no longer sufficient for the future.

The future

As we move to AWS, we are changing the structure of the underlying platform to ensure it can grow to handle millions of organisations. The new platform is structured around cells. Each cell contains everything needed to run the Xero application for 100,000 subscriptions: all of the networks, storage, servers, databases and application software.

the future

In many ways, cells are the logical extension of the sharding method that has worked so well at Xero. Previously, using shards, we gave groups of customers their own database clusters, but used shared infrastructure for everything else. With cells, groups of 100,000 subscriptions run on their own dedicated infrastructure, including networks, storage, servers, databases and application software.

The central reason we’re adopting the cell model is to allow us to continue to grow the platform without compromising service or performance as we add customers. Cells provide some other important benefits as well:

  • They enable smarter ways of deploying software. Using cells we can deploy new features to a subset of organisations and test two different versions of the same feature at once to see which works best.
  • They limit the impact of a failure. Although it is impossible to completely avoid infrastructure failures, with cells, we can control how much impact those failures have and how quickly we respond to them.

As a customer, you probably won’t notice the change in the underlying structure of Xero’s platform but over time it will allow us to grow and support millions of subscribers.

Stay tuned for more information around the technical architecture of Xero’s new platform and why we’re making these changes.

5 comments

Heather Smith
July 13, 2016 at 4.51 am

Mark,

That was a great explanation – even a dummy like me thinks I understand what’s happening here. The ‘platform’ always seems to be pivotal to growth of the business and use of the solution.

As a long term customer my internet speed has increased – but I find accessing Xero has slowed down – so I do hope I see an improvement in performance!

As the cells – have their own infrastructure support – could we start to see data housed in different countries? Could we set up a Singapore client and opt for them to be housed in a Singapore cell? Surely infrastructure closer to the location will impact speed? Plus minimise issues around the location of data storage?

I understand data storage is moving to Amazon? There is Amazon data storage here in Australia? Why are some of the Xero subscriptions not spread across the world and stored there?

As an advisor are all my subscriptions I manage stored in a single share or spread all over the place?

Cheers

Heather Smith

Duncan Ritchie in reply to Heather Smith Xero
July 15, 2016 at 5.08 pm

Hi Heather

We have a few areas where users have had slow performance. We have work underway to resolve that alongside the migration to AWS.

While our cell model gives us the base capabilities to host customers in multiple countries we still have a number of things to do to support separation across countries. We have done some planning on how we could do this but we do not have any firm plans to have data and hosting outside of the US.

All of your subscriptions are in one AWS region however they will be separated across cells.

Cheers
Duncan

Di Crawford-Errington
July 13, 2016 at 6.37 am

Awesome explanation Mark. That totally makes sense even to me. Thank you. As Heather said hopefully this will improve the speed. I look forward to seeing the results.

Greg Henderson
July 15, 2016 at 9.24 am

Glad to see a tech company thinking about the future and keeping the customer experience good. In another blog post https://www.xero.com/blog/2016/05/were-improving-infrastructure-platform-aws/ Duncan Ritchie mentions that the cells will be hosted on AWS in US data centres. I am in Australia. Have you considered the effect of network latency in this design? AWS do have a presence in Australia. Have you considered hosting your Australian customers there?

Duncan Ritchie in reply to Greg Henderson Xero
July 15, 2016 at 5.12 pm

Hi Greg

Yes we have considered having regional hosting closer to each customer. At this point we have no plans to make this change however we will continue to review this decision.

We do have a robust content delivery network (CDN) in place that serves around 9 in 10 requests from a local server. Often these servers are quite close to you as our provider has a few hundred thousand servers around the world.

Cheers
Duncan

Leave a reply

Your email address will not be published. Required fields are marked *