Apple has open sourced its distributed database core, FoundationDB, which it acquired back in 2015 from the company of the same name. FoundationDB is designed to handle large volumes of data stored across clusters of commodity servers, and to favor data consistency by supporting fully global, cross-row ACID transactions.
FoundationDB, as its name implies, provides a low-level foundational interface for systems that need to store data by preserving consistency and being able to scale. The basic idea behind FoundationDB is to provide a datastore "core" that exposes an ordered key-value store with transactions, on top of which multiple database front-ends with different data models and query languages can be built. This includes, for example, an SQL database, a document database, a graph database, or more specialized APIs, such as Lucene/ElasticSearch, Zookeeper, and so on. All of them would use the underlying datastore core to ensure scalability and consistency of operation.
According to Apple, FoundationDB scales well as you add machines to the clusters, which are additionally able to automatically heal from hardware failures.
One of the founders of FoundationDB, the company that originally developed the FoundationDB datastore, provided a few more details about its design on Hacker News. FoundationDB uses optimistic concurrency, and handles the possibility of conflicts through lexicographic conflict ranges rather than using locks. Each range is an interval of one or more keys read or written by a transaction, with minimum granularity of a single key. Conflict ranges intersections are resolved by a set of internal microservices which partition up the key space independently of the way it is partitioned for data storage.
Furthermore, to improve scalability, FoundationDB represents indexes directly in the key/value store and keeps them consistent with the data through transactions. The benefit of this approach is it does not require the broadcast of indexes to all nodes. Another interesting feature of FoundationDB is strict serializable isolation with external consistency for all transactions.
Developers can interact with FoundationDB using API bindings available for a number of languages, including C, Go, Java, Python, and Ruby. Binary installers are available for macOS, Windows, and Linux. While the macOS platform is fully supported for development purposes, it is useful to keep in mind that production use is only supported by the Linux version.
