In this interview from QCon San Francisco 2007, Randy Shoup discusses the architecture of eBay. Topics discussed include eBay's architectural principles, horizontal and vertical partitioning, ACID vs. BASE, handling data inconsistency, distributed caching, updating eBay on the fly, architectural and coding standards, eBay's search infrastructure, grid computing, and SOA.
Watch Randy Shoup Discusses the eBay Architecture (38 minutes).
When asked in what design areas eBay goes against the grain of current thinking, Shoup said:
Partitioning is a first example. We partition for scale, we partition to make sure that we take the huge amount of load and users that is the overall eBay problem and break it down into smaller manageable chunks of that problem: chunks of users, chunks of items, chunks of traffic. The way that that makes us go against the grain is that… Well, I'll explain a little bit about how we partition the databases. So, we don't have a single database that runs the entire site, we have many different logical databases that are each different functional parts of the site. So you can imagine a database that has selling information, a database that has item information, user information, transactions that have occurred and so on.
We actually have a thousand different logical database instances spread over four hundred servers. Within those individual functional, or vertical partitions if you like, we do horizontal partitioning by the data. So I told you that there is one logical item database you can imagine, but in actuality there are twenty individual item databases where a given item lives on one of those twenty, and each of them has one-twentieth of the overall number of items. And we do the same thing for users, we do the same thing for transactions, accounts, that's a pattern we follow everywhere. So how does that make us break the rules, is that whenever we need to do a particular operation -- almost every operation is touching items, let's say users accounts; some multiple different functional partitions or vertical partitions.
If we did it in the standard way we'd do a distributed transaction with two phase commit across all those different hosts. Because for scale we have to split up the architecture in those individual chunks, again if we did it in a transactional way it would force us to do distributed transactions essentially all the time. And for very common but very expensive use cases like selling a new item -- in other words adding a new item to the site -- we might literally be touching twenty or thirty different hosts. So imagine a two-phase commit across twenty or thirty instances. That wouldn't work. So how do we break the rules? We break the rules by not doing a transaction. "Doctor, doctor, it hurts when I go like this". "Don't go like that". So what we follow is, rather than an ACID style, something that we call the BASE style, you probably would have heard that as well. Basically Available, Soft-state, Eventually consistent. So we commit as much as we can to individual hosts and then if there are inconsistencies that need to be reconciled we reconcile those through some kind of asynchronous process.