Confluent recently released Stream Designer, a visual interface that lets developers quickly build and deploy streaming data pipelines.
The company aims to simplify data streaming pipelines with a point-and-click visual builder with Stream Designer. The company describes Stream Designer as a service accessible to developers beyond specialized Apache Kafka experts in a press release.
Stream Designer allows users to build pipelines from a unified view in Confluent Cloud and take advantage of built-in integrations with fully-managed connectors, ksqlDB, and Kafka topics. In addition, it provides integration with the users' existing DevOps practices and allows them to switch between the graphical canvas and a complete SQL editor.
Source: https://www.confluent.io/blog/building-streaming-data-pipelines-visually/
Jay Kreps, co-founder and CEO of Confluent, said in the same press release:
We are in the middle of a major technological shift, where data streaming is making real-time the new normal, enabling new business models, better customer experiences, and more efficient operations. With Stream Designer, we want to democratize this movement toward data streaming and make real-time the default for all data flow in an organization.
To learn more about the Stream Designer, InfoQ reached out to Jon Fancey, a principal product manager who acted as product owner for the new service.
InfoQ: Where did the Stream Designer come from? Was this an internal product to become a public offering, or was this created from the start to be customer-facing?
Jon Fancy: Many companies are building data pipelines with Kafka today, and many of them face the same challenges: Kafka expertise shortages and taxing development cycles. We wanted to make data streaming accessible to organizations that cannot afford huge Kafka teams and developers with various skill sets. It was created from the start to be customer-facing.
InfoQ: What are the clear use cases for the Stream Designer?
Fancy: Stream Designer can be used to make building anything on Kafka and Confluent Cloud easier and faster. For example, it’s excellent for building, testing, and deploying streaming data pipelines for operational and analytical use cases and turning traditional ETL pipelines that serve in those areas into a real-time paradigm.
InfoQ: How do users interact with the Stream Designer? What's the UX?
Fancy: Users can interact with Stream Designer through the browser-based visual canvas and the built-in SQL editor - whatever method works best for them. They can point and click to add pipeline components on the canvas and configure them from the right panel. Users can also drag and drop connections between components to create joins and forks. Pipeline definitions are translated to SQL under the hood and can also be extended with code. Stream Designer also allows users to import pre-built pipelines as code, such as recipes, for teams to use as templates.
InfoQ: What are comparable products in the markets? Or is it unique?
Fancy: Stream Designer is the industry’s first visual interface for building streaming pipelines natively on Kafka. There are undoubtedly other GUI-based stream processors out there, but other messaging technologies power them. Stream Designer is built on Kafka, the de facto standard for data streaming and the developer’s open platform of choice with high scalability and resiliency.
InfoQ: How does pricing now work for the Stream Designer?
Fancy: There is no extra cost for Confluent Cloud customers.
InfoQ: What is the roadmap for the future of Stream Designer?
Fancy: My philosophy with any product is that GA is just the beginning. We plan to further integrate Stream Designer with our platform and the streaming ecosystem and listen to customer feedback to guide what we iterate on next.
