InfoQ Homepage Event Stream Processing Content on InfoQ
-
/articles/kafka-clusters-cloudflare/en/smallimage/tales-of-Kafka-at-Cloudflare-small-1684768928637.jpg)
Tales of Kafka at Cloudflare: Lessons Learnt on the Way to 1 Trillion Messages
Cloudflare uses Kafka clusters to decouple microservices and communicate the creation, change or deletion of various resources via protobuf, a common data format in a fault-tolerant manner. The authors suggest investing in metrics for problem detection, prioritizing clear SDK documentation, and balancing flexibility and simplicity for standardized pipelines.
-
/articles/billions-messages-minute/en/smallimage/LOGO-1679471830613.jpg)
Billions of Messages Per Minute Over TCP/IP
Chronicle Wire offers an alternative way of transferring data between systems, delivering more messages, faster, than common JSON/XML approaches. This approach to data serialization improves both latency and throughput.
-
/articles/streaming-first-real-time-ml/en/smallimage/streaming-ml-small-logo-1660833205193.jpeg)
Streaming-First Infrastructure for Real-Time Machine Learning
This article covers the benefits of streaming-first infrastructure for two scenarios of real-time ML: online prediction, where a model can receive a request and make predictions as soon as the request arrives, and continual learning, when machine learning models are capable of continually adapting to change in data distributions in production.
-
/articles/network-proxy-stream-processor-pipy/en/smallimage/logo-1643366806668.jpg)
How to Create a Network Proxy Using Stream Processor Pipy
In this article we are going to introduce Pipy, an open-source cloud-native network stream processor. After describing its modular design, we will see how to rapidly build a high-performance network proxy to serve our specific needs. Pipy has been battle-tested and is already in use by multiple commercial clients.
-
/articles/whats-the-next-step-for-data-management/en/smallimage/database-1605276684430.jpeg)
Beyond the Database, and beyond the Stream Processor: What's the Next Step for Data Management?
Databases have been around forever with the same shape: you make a request to your data and then you receive an answer. Now, stream processors came along with a different approach: data isn’t locked up, it is in motion. Understand how stream processors and databases relate and why there is an emerging new category of databases that focus on data that stays in place as well as data that moves.
-
/articles/real-time-api-kafka/en/smallimage/real-time-api-kafka-s-1602600487985.jpeg)
Real Time APIs in the Context of Apache Kafka
Events offer a Goldilocks-style approach in which real-time APIs can be used as the foundation for applications which is flexible yet performant; loosely-coupled yet efficient. Apache Kafka offers a scalable event streaming platform with which you can build applications around the powerful concept of events.
-
/articles/realtime-event-driven-ecosystem/en/smallimage/realtime-api-logo-1595963690562.jpg)
The Challenges of Building a Reliable Real-Time Event-Driven Ecosystem
Globally, there is an increasing appetite for data delivered in real time; we are witnessing the emergence of the real time API. When it comes to event-driven APIs engineers can choose between multiple different protocols. In addition to choosing a protocol, engineers also have to think about subscription models, too: server-initiated (push-based) or client-initiated (pull-based).
-
/articles/applied-probability-unstructured-events-theta-sketches/en/smallimage/applied-probability-unstructured-events-theta-sketches-s-1593199758266.jpg)
Applied Probability - Counting Large Set of Unstructured Events with Theta Sketches
In this article, author Ronen Cohen discusses the solution to processing the event data using Theta Sketches and technologies like HBase and Kafka.
-
/articles/edge-computing-streaming-data/en/smallimage/edge-computing-streaming-data-s-1586865838285.jpg)
Is Edge Computing a Thing?
Edge Computing is definitely a thing, but the computing need not occur at the edge. Instead what is needed is an ability to compute (anywhere) on streaming data from large numbers of dynamically changing devices, in the edge environment. This in turn demands an architectural pattern for stateful, distributed computing.
-
/articles/building-scalable-iot-application-with-apache-kafka/en/smallimage/The-Kongo-Problem-Building-a-Scalable-IoT-Application-with-Apache-Kafka-logo-1581595522263.jpg)
The Kongo Problem: Building a Scalable IoT Application with Apache Kafka
In this article, author Paul Brebner discusses the best practices for developing IoT projects using Apache Kafka and Kafka Streams technologies and how to maximize Kafka scalability.
-
/articles/rethinking-flink-api/en/smallimage/Rethinking-Flinks-APIs-logo-1568358627523.jpg)
Rethinking Flink’s APIs for a Unified Data Processing Framework
Since its very early days, Apache Flink has followed the philosophy of taking a unified approach to batch and streaming. The core building block is the “continuous processing of unbounded data streams, with batch as a special, bounded set of those streams.” Recent updates to the Flink APIs include architectural designs by the community to support batch and streaming unification in Apache Flink.
-
/articles/patient-care-stream-processing/en/smallimage/patient-care-stream-processing-s-1547626914062.jpeg)
Increasing the Quality of Patient Care through Stream Processing
Today’s healthcare technology landscape is disaggregated and siloed. Physicians analyse patient data streams from different systems without much correlation. Even though health-tech domain is mature and rich with data, the value of it is not directed towards increasing the quality of patient care. This article presents a stream processing solution in which streams are co-related.