InfoQ Homepage Apache Kafka Content on InfoQ
-
/articles/managing-memberships-netflix/en/smallimage/Managing238M-1710832361735.jpg)
Managing 238M Memberships at Netflix
In this article Surabhi Diwan shared how the Netflix membership team does distributed systems: the architecture bets, technology choices, and operational semantics that serve the needs of Netflix’s ever-growing member base.
-
/articles/kafkaflow-dotnet-framework/en/smallimage/building-kafka-event-driven-applications-with-KafkaFlow-small-1693903734363.jpg)
Building Kafka Event-Driven Applications with KafkaFlow
KafkaFlow, a .NET open-source project, simplifies Kafka-based event-driven app development with features like middleware for message processing, enhancing maintainability, customization potential, and allowing developers to prioritize business logic.
-
/articles/kafka-clusters-cloudflare/en/smallimage/tales-of-Kafka-at-Cloudflare-small-1684768928637.jpg)
Tales of Kafka at Cloudflare: Lessons Learnt on the Way to 1 Trillion Messages
Cloudflare uses Kafka clusters to decouple microservices and communicate the creation, change or deletion of various resources via protobuf, a common data format in a fault-tolerant manner. The authors suggest investing in metrics for problem detection, prioritizing clear SDK documentation, and balancing flexibility and simplicity for standardized pipelines.
-
/articles/series-apache-kafka-streams/en/smallimage/apache-kafka-article-series-logo-small-1673455781546.jpg)
Article Series: Developing Apache Kafka applications on Kubernetes
Apache Kafka has integrations with most of the languages used these days, but in this article series, we cover its integration with Java. In this series, we also discuss how to provision, configure and secure an Apache Kafka cluster on a Kubernetes cluster.
-
/articles/secure-kafka-cluster-strimzi/en/smallimage/securing-a-kafka-cluster-in-kubernetes-using-strimzi-small-1671647644310.jpg)
Securing a Kafka Cluster in Kubernetes Using Strimzi
Deploying an Apache Kafka cluster to Kubernetes is easy if you use Strimzi, but that’s only the first step; you need to secure the communication between Kafka and the consumers and producers, provide RBAC to access topics, spread the secrets correctly to Kafka Connect components and all using a Kubernetes GitOps way.
-
/articles/high-fidelity-data-streams/en/smallimage/building-and-operating-high-fidelity-data-streams-small-logo-1664284454024.jpg)
Building & Operating High-Fidelity Data Streams
At QCon Plus 2021 last November, Sid Anand, chief architect at Datazoom and PMC Member at Apache Airflow, presented on building high-fidelity nearline data streams as a service within a lean team. In this talk, Anand provides a master class on building high-fidelity data streams from the ground up.
-
/articles/strimzi-the-gitops-way/en/smallimage/strimzi-gitops-small-logo-1664297048552.jpg)
Moving Kafka and Debezium to Kubernetes Using Strimzi - the GitOps Way
Deploying an Apache Kafka cluster to a Kubernetes is not an easy task. There are a lot of pieces to configure like the zookeeper, the Kafka cluster, topics, and users. Strimzi is a Kubernetes controller making the deployment process of Kafka a child game. Moreover, Strimzi lets you manage Kafka using GitOps methodology as everything is executed using a Kubernetes YAML file.
-
/articles/change-data-capture-debezium/en/smallimage/logo-1660294927422.jpg)
Debezium and Quarkus: Change Data Capture Patterns to Avoid Dual-Writes Problems
It’s common in microservices to write data in two places, a database and then send the content to another microservice. One approach to tackle this problem is dual writes, but you may lose data because of concurrent writes. Debezium is an open-source project for change data capture using the log scanner approach to avoid dual writes and communicate persisted data correctly between services.
-
/articles/quarkus-with-kafka-streams/en/smallimage/kafka-streams-and-quarkus-real-time-processing-events-small-1656083680635.jpg)
Kafka Streams and Quarkus: Real-Time Processing Events
Consuming Kafka messages is simple; you get them as long as they are produced, but nothing more. But if you need real-time processing of the data (filtering, joining, or manipulating events), just using the Kafka-consuming API might not be the best approach as the resulting code becomes complex. Kafka Streams and Quarkus are the perfect matches to start processing Kafka events in real-time.
-
/articles/data-with-quarkus-kafka/en/smallimage/UaUXvhM4-1651071035514.jpg)
Getting Started to Quarkus Reactive Messaging with Apache Kafka
How data is processed/consumed nowadays is different from how it was once practiced. In the past, data was stored in a database and it was batch processed for analytics. Apache Kafka is a distributed event store and stream-processing platform for storing, consuming, and processing data streams in real-time. In this post, we’ll learn how to produce and consume data using Apache Kafka and Quarkus.
-
/articles/microservices-inside-out/en/smallimage/turning-microservices-inside-out-small-image-1627634447180.jpg)
Turning Microservices Inside-Out
Turning microservices inside-out means moving past a single, request/response API to designing microservices with an inbound API for queries and commands, an outbound APIs to emit events, and a meta API to describe them both. A database can be supplemented with Apache Kafka via a connecting tissue such as Debezium.
-
/articles/database-audit-system-kafka/en/smallimage/Building-a-SQL-Database-Audit-System-Using-Maxwell-logo-small-1609154420148.jpg)
Building an SQL Database Audit System Using Kafka, MongoDB and Maxwell's Daemon
In this article, the author discusses the importance of a database audit logging system outside of traditional built-in data replication, using technologies like Kafka, MongoDB, and Maxwell's Daemon.