The recent theft from several BitCoin operators has sparked a debate whether eventually consistent databases are useful or not for banking.
On March 2, 2014 Flexcoin lost all its bitcoins due to a code flaw. The attacker issued thousands of concurrent requests ordering transfers from one of his accounts to another. He then repeated the operation with other accounts until all bitcoins were withdrawn. This was possible because the code was not written to deal with multiple concurrent requests, and all the transfers happened before the balances were updated. If a balance is not updated in time, a new request could be granted even if the account normally is empty. As a result, Flexcoin shut down operations after losing 896 BTC valued at about half a million dollars.
The same happened with Poloniex two days later, but they lost only 12.3% of their bitcoins and the company covered the losses, managing to stay afloat.
Emin Gün Sirer, an Associate Professor at Cornell University, wrote a blog post blaming eventually consistent data stores for the lost bitcoins. He mentions MongoDB, Cassandra and Riak among the NoSQL solutions that are vulnerable to banking thefts because:
The problem here stemmed from the broken-by-design interface and semantics offered by MongoDB. And the situation would not have been any different if we had used Cassandra or Riak. …
Bitcoin coincided with a particularly dark time in distributed systems when people, armed with an incorrect interpretation of the CAP Theorem, thought that they just had to give up on consistency in their databases …
It is not clear if Flexcoin or Poloniex were using MongoDB at that time, and it should be mentioned that Sirer is involved in developing HyperDex, a competing key-value data store supporting ACID transactions. Also, this is not the first time Sirer blames MongoDB’s design. A year ago, he claimed that MongoDB’s fault tolerance is broken.
Polemics aside, it is worth pondering if an eventually consistent data store is suitable for banking operations. As expected, Sirer’s post triggered a large number of comments, and we extracted some of the most significant ones.
jrp noted that the update operation could have been done with MongoDB at the database level in an atomic way, but agreed that “this will not take care of other ACID properties.”
jakcharlton considered that while eventual consistency is problematic in such cases, an “ACID system does not solve the problem, it merely pushes it to an application boundary, where the problem re-occurs. Banking was a bad example domain to use, and shows the issue of developers thinking that as they have ACID in their database they are protected from issues of concurrency.”
Alex uses MongoDB for “everything except where transactions are needed and then I use the RIGHT tool for the job (which is not MongoDB).” He considered that the developer made a mistake by using MongoDB for a job that it should not be used for.
Robert Escriva, a HyperDex developer, considered that the culprit is not with the programmer but a general perception that eventually consistent systems can be used for banking:
The problem is not with programmers' understanding. There's a pervasive and wrong movement that encourages people to use eventually consistent systems. They will justify it with statements of the form, "If it's good enough for banks, then it's good enough for you." This thinking is dangerous.
At the end of the day, your application is the sum of its invariants. If the data store underneath your system is incapable of providing you with guarantees---or worse, requires extensive care, and a $100K support contract to provide you with guarantees---your application is broken and blaming the developers for this is a cop-out. We should be holding database vendors (especially those with hundreds of million in shiny VC cash) to a higher standard.
Eric Brewer, author of the CAP theorem, wrote earlier that banks give to consistency for their ATM operations in order to provide availability during partitioning. But they do this by taking certain precautions, including limiting the amount of money withdrawn at a certain small threshold. We should also mention that Stripe, a web&mobile payment system uses MongoDB, according to one of their blog posts.
Are eventually consistent data stores appropriate for general banking operations or should developers be aware of their limitations and look elsewhere?