InfoQ Homepage Streaming Content on InfoQ
-
/articles/applied-probability-unstructured-events-theta-sketches/en/smallimage/applied-probability-unstructured-events-theta-sketches-s-1593199758266.jpg)
Applied Probability - Counting Large Set of Unstructured Events with Theta Sketches
In this article, author Ronen Cohen discusses the solution to processing the event data using Theta Sketches and technologies like HBase and Kafka.
-
/articles/edge-computing-streaming-data/en/smallimage/edge-computing-streaming-data-s-1586865838285.jpg)
Is Edge Computing a Thing?
Edge Computing is definitely a thing, but the computing need not occur at the edge. Instead what is needed is an ability to compute (anywhere) on streaming data from large numbers of dynamically changing devices, in the edge environment. This in turn demands an architectural pattern for stateful, distributed computing.
-
/articles/building-scalable-iot-application-with-apache-kafka/en/smallimage/The-Kongo-Problem-Building-a-Scalable-IoT-Application-with-Apache-Kafka-logo-1581595522263.jpg)
The Kongo Problem: Building a Scalable IoT Application with Apache Kafka
In this article, author Paul Brebner discusses the best practices for developing IoT projects using Apache Kafka and Kafka Streams technologies and how to maximize Kafka scalability.
-
/articles/rethinking-flink-api/en/smallimage/Rethinking-Flinks-APIs-logo-1568358627523.jpg)
Rethinking Flink’s APIs for a Unified Data Processing Framework
Since its very early days, Apache Flink has followed the philosophy of taking a unified approach to batch and streaming. The core building block is the “continuous processing of unbounded data streams, with batch as a special, bounded set of those streams.” Recent updates to the Flink APIs include architectural designs by the community to support batch and streaming unification in Apache Flink.
-
/articles/azure-data-lake-analytics-usql/en/smallimage/azure-data-logo-1568103168665.jpg)
Azure Data Lake Analytics and U-SQL
In this article, the author shows how to use big data query and processing language U-SQL on Azure Data Lake Analytics platform. U-SQL combines the concepts and constructs both of SQL and C#. It combines the simplicity and declarative nature of SQL with the programmatic power of C# including rich types and expressions.
-
/articles/anamoly-detection-yurita/en/smallimage/logo-1562313713564.jpg)
Stream Processing Anomaly Detection Using Yurita Framework
In this article, author Guy Gerson discusses the stream processing anomaly detection framework they developed by PayPal, called Yurita. The framework is based on Spark Structured Streaming.
-
/articles/prometheus-monitor-applications-at-scale/en/smallimage/logo-1560849701397.jpg)
How to Use Open Source Prometheus to Monitor Applications at Scale
In this article, the author discusses how to collect metrics and achieve anomaly detection from streaming data using Prometheus, Apache Kafka and Apache Cassandra technologies.
-
/articles/data-processing-redis-spark-streaming/en/smallimage/real-time-data-processing-logo-small-1557414118420.jpg)
Real-Time Data Processing Using Redis Streams and Apache Spark Structured Streaming
Structured Streaming, introduced with Apache Spark 2.0, delivers a SQL-like interface for streaming data. Redis Streams enables Redis to consume, hold and distribute streaming data between multiple producers and consumers. In this article, author Roshan Kumar walks us through how to process streaming data in real time using Redis and Apache Spark Streaming technologies.
-
/articles/patient-care-stream-processing/en/smallimage/patient-care-stream-processing-s-1547626914062.jpeg)
Increasing the Quality of Patient Care through Stream Processing
Today’s healthcare technology landscape is disaggregated and siloed. Physicians analyse patient data streams from different systems without much correlation. Even though health-tech domain is mature and rich with data, the value of it is not directed towards increasing the quality of patient care. This article presents a stream processing solution in which streams are co-related.
-
/articles/distributed-stream-processor-container/en/smallimage/distributed-stream-processor-container-s-1547155264550.jpeg)
Scaling a Distributed Stream Processor in a Containerized Environment
The article presents our experience of scaling a distributed stream processor in Kubernetes. The stream processor should provide support for maintaining the optimal level of parallelism. However, adding more resources incurs additional cost and also it does not guarantee performance improvements. Instead, the stream processor should identify the level of resource requirement and scale accordingly.
-
/articles/apache-kafka-best-practices-to-optimize-your-deployment/en/smallimage/apache-kafka-best-practices-to-optimize-your-deployment-s-1539792752614.jpeg)
Apache Kafka: Ten Best Practices to Optimize Your Deployment
Author Ben Bromhead discusses the latest Kafka best practices for developers to manage the data streaming platform more effectively. Best practices include log configuration, proper hardware usage, Zookeeper configuration, replication factor, and partition count.
-
/articles/democratizing-stream-processing-kafka-ksql-part2/en/smallimage/Democratizing-Stream-Processing-Apache-Kafka-KSQL-Part-2-logo-small-1536080637584.jpg)
Democratizing Stream Processing with Apache Kafka® and KSQL - Part 2
In this article, author Robin Moffatt shows how to use Apache Kafka and KSQL to build data integration and processing applications with the help of an e-commerce sample application. Three use cases discussed: customer operations, operational dashboard, and ad-hoc analytics.