Soon to enter beta, D1 is Cloudflare's first step into the Cloud-based SQL storage arena. D1 is built on top of SQLite with the addition of a distributed replication mechanism, batch operation support, embedded compute, automatic backups and redundancy, and more.
Cloudflare says D1 is aimed to integrate seamlessly with Cloudflare Workers, which provide a serverless execution environment that runs across Cloudflare edge infrastructure. The key point to understand the value of Cloudflare announcement lies with the combination of SQL with "serverless".
In fact, Cloudflare already provides a number of serverless storage options for Workers, including Durable Objects, which are actor-like objects with attached persistent storage, KV, which provides a key-value store, and R2, an S3-compatible storage service providing direct object storage.
Until now, though, Cloudflare was not providing a complete relational storage option. Indeed, previous to D1, Cloudflare's only support for relational storage was through Relational Database Connectors, which provided an easy way to connect Workers to a relational database but did not include the actual storage service. In other words, the relational database had still to run outside of Cloudflare infrastructure and within the customers' Cloud environment.
D1 fills thus the niche of applications requiring SQL without being so complex as to justify operating a full MySQL or PostgreSQL deployment. Essential to this is leveraging SQLite as the engine for relational storage without involving a specific server to run a full relational database.
Since Workers itself runs between the server and the client, and was inspired by technology built for the client, SQLite seemed like the perfect fit for our first entry into databases. [...] the strength of D1 is the developer experience: allowing you to go from nothing to a full stack application in an instant.
To make this possible, Cloudflare has built a replication mechanism around SQLite box that will create read-only clones of the database close to users by leveraging Cloudflare Edge infrastructure, and ensure all of them are kept in sync when changes happen.
Cloudflare has not released yet detail about how this replication mechanism will work. Cloudflare engineer greg-m pointed out on Hacker News that D1 will not provide strong consistency out of the box if read replicas are used. This leads to think that a background process will be in charge of ensuring all the replicas are eventually synchronized.
As a comparison, rqlite an open source distributed relational database using SQLite, uses the Raft consensus algorithm that uses a leader to reach consensus. Any change will only be acknowledged when a quorum of nodes has persisted it. The cost of this approach is increased commit latency, but with a strong consistency guarantee.
D1 will also support batch operations through its API to make it possible to execute a list of SQL statements in a single HTTP request, which can help building atomic operations. Also supported are automatic snapshotting to R2 and redundancy leveraging Durable Objects.
Another interesting capability in D1 is embedded compute, which makes it possible to hand off a specific request received by a given Worker close to the user to another Worker associated with the primary D1 instance to complete the operation.
Cloudflare D1 provides a Web-based UI to create and manage a database as well as a CLI-first option using Wrangler 2.
