Debezium is an open source change data capture (CDC) tool built on top of Kafka that captures and publishes changes made in a database as a stream of events. Debezium 1.0 Final was recently released with event format clean-up, increased test coverage of databases, including Postgres 12, SQL Server 2019 and MongoDB 4.2, as well as a number of bug fixes with 96 issues addressed in version 1.0 and the preview releases. In a blog post Gunnar Morling describes Debezium's basic concepts and some common use cases, and details about both the current release and what to expect in future releases.
For Morling, software engineer at Red Hat and project lead of Debezium, what's most important in the new release is the event format clean-up. They have increased their commitment to keep the emitted event structures and configuration options of the connectors correct and consistent, and to make sure these are evolved in a backwards-compatible way. But he also points out that they have never broken things before.
In an interview with InfoQ, Morling notes some features added to the 0.10 release that he thinks are also relevant to mention for the current 1.0 release. For Postgres 10, they now support the pgoutput logical decoding plug-in and exported snapshots with no locks required. An incubating connector for Cassandra, extended and more unified metrics across the different connectors, and customizable message keys are other new features.
Morling emphasizes that the 1.0 Final release is the result of work of the Debezium community at large. Led by Red Hat, about 150 people have contributed to the project, and he thinks this is great for an open source project. He also points out the importance of users sharing their experiences in conference talks and blog posts because hearing about real usage of CDC and Debezium helps the team improving the product.
Upgrading from the earlier 0.10 release is mainly a drop-in replacement. For earlier releases, there are migration notes available describing upgrading procedures and deprecated options.
The work with Debezium 1.1 has started with 1.1.0 Alpha1 just released. One new feature is the Quarkus outbox pattern extension that will simplify the creation of outbox events within an application. Debezium supports the outbox pattern for integration of microservices and the team wants to improve on this with the Quarkus extension that complements the existing event router for events from an outbox table.
Other features added or considered for the new release include incubating support for CloudEvents and IBM DB2, exposing topics with transaction events, and adding SPI for customizing schema and value representation of given columns. A standalone container for running Debezium without Apache Kafka and Connect is also on the roadmap.
Red Hat is working on a commercially supported offering around CDC. In preparation for a GA release, four connectors are currently available as a Technology Preview. These are part of the Red Hat Integration product, targeting deployments on OpenShift via the AMQ Streams Kafka operator.
In a podcast recently published on InfoQ, Wesley Reisz talks with Morling about the Debezium project, CDC, and some of the use cases. They also discuss the long-term strategic goals for the project.
At the microXchg 2019 conference in Berlin, Morling discussed how CDC can be used to create events from a database in a microservices architecture.