Confluent recently announced new enhancements to its Stream Governance product that will improve engineering teams' ability to discover, understand, and trust real-time data. Organizations can use Stream Governance Advanced to resolve issues within complex pipelines more easily with point-in-time lineage, discover and understand topics more quickly with business metadata, and enforce quality controls globally with Schema Registry.
Chad Verbowski, senior vice president of engineering, Confluent, said in a press release:
Businesses heavily rely on real-time data to make fast and informed decisions, so it’s paramount that the right teams have quick access to trustworthy data. With Stream Governance, organizations can understand the full scope of streams flowing across their business so they can quickly leverage that data to power endless use cases.
To learn more about Stream Governance Advanced, InfoQ reached out to David Araujo, principal product manager at Confluent.
InfoQ: Where did the Stream Governance Advanced tier come from? Was this created from customer feedback?
David Araujo: Stream Governance Advanced was devised following the launch of our governance suite in September 2021. Wide adoption of Stream Governance features proved this to be a critical need for businesses. Still, customer feedback guided us toward where and how we needed to continue evolving the product. Customers seeking to expand their use of Apache Kafka and real-time data streaming for even more sophisticated, mission-critical use cases were dependent upon the quality and visibility/discovery tools that could scale enterprise deployment.
InfoQ: What are the clear use cases for the Stream Governance Advanced tier?
Araujo: Stream Governance Advanced delivers enterprise-grade governance and data visibility for production workloads, allowing businesses to:
- Confidently govern mission-critical workloads at any scale with a new 99.95% uptime SLA for Schema Registry available across 28 global regions (Stream Quality)
- The 99.95% uptime SLA for Schema Registry (new for Advanced) allows teams to offload more workloads to the cloud with high confidence that data quality controls for Apache Kafka will be highly available—especially valuable for teams self-managing open-source Kafka deployments who can shift to a highly available, fully managed service.
- Schema Registry support across 28 global regions (expanded for Advanced) allows teams to optimize performance on the data streaming platform and further establish data sovereignty with schemas sitting closer to their corresponding Kafka clusters or any available region of choice.
- Enhance data discovery within your streaming catalog with user-generated business context and easy, declarative search via GraphQL API (Stream Catalog)
- Business metadata (new for Advanced) allows individual users to add custom, open-form details to platform objects such as a topic in order to help other users understand which team owns the topic, how it is being used, who to contact with questions about the data, or any other details they deem necessary.
- The GraphQL API (new for Advanced) allows users to take advantage of the graph nature of the Stream Catalog, which is modeled as a graph of entities and relationships, and provides them with a more natural, efficient, and productive way of programmatically exploring the catalog.
- Simplify comprehension and troubleshooting of complex data streams with lineage search and historical point-in-time insights (Stream Lineage)
- Point-in-time lineage (new for Advanced) provides users with a look into the past—allowing them to see data stream evolutions over 24 hours or within any 1-hour window over a 7-day range in order to answer questions such as, "What happened to the pipeline on Friday at 3 pm when support tickets started arriving?" or "What did this pipeline look like last week when my manager seemed happier with the configuration?"
- Search the lineage graph (new for Advanced) allows users to save time during development or investigations by finding specific objects such as client IDs or topics buried with complex data pipelines.
InfoQ: Is there complete feature parity between the new advanced tier and the essentials tier? What are the differences?
Araujo: All features included within the Essentials tier are included within Advanced. Stream Governance Advanced introduces net-new features for stream catalog and stream lineage not found in Essentials alongside higher limits, expanded regional coverage, and an increased SLA for Schema Registry (stream quality). Full side-by-side details here.
InfoQ: How does pricing now work for the Stream Governance offering?
Araujo: Stream Governance Essentials is made available to all Confluent Cloud customers free of charge for up to 1,000 schemas per environment, after which schemas are billed at a rate of $0.002/schema/hour.
Stream Governance Advanced is priced at $1/hour/environment with support for up to 20,000 total schemas per environment.
InfoQ: What is the roadmap for the future of Stream Governance?
Araujo: Since launching in 2021, Stream Governance has become a critical suite of capabilities for Confluent customers seeking to safely expand data streaming deployments in the cloud to deliver against growing customer expectations for "real-time everything." As such, we continue to invest heavily in developing more governance features across Confluent’s entire data streaming platform to simplify further customer efforts to move real-time data throughout their entire tech stack and achieve that goal.
