InfoQ Homepage Streaming Content on InfoQ
-
Stream All the Things: Patterns of Effective Data Stream Processing Explored by Adi Polak at QCon SF
Adi Polak, Director of Advocacy and Developer Experience Engineering at Confluent, illuminated the complexities of data streaming in her QCon San Francisco presentation. She outlined key design patterns for robust pipelines, emphasizing reliability, scalability, and data integrity.
-
MLow: Meta Introduces Audio Codec for Low-End Devices
Meta has recently announced the Meta Low Bitrate (MLow) audio codec, aimed at enhancing real-time communication (RTC) on low-end devices and slow-speed connections. This new codec is designed to deliver efficient, high-quality audio streaming, even under low bandwidth conditions.
-
Yelp Overhauls Its Streaming Architecture with Apache Beam and Apache Flink
Yelp reworked its data streaming architecture by employing Apache Beam and Apache Flink. The company replaced a fragmented set of data pipelines for streaming transactional data into its analytical systems, like Amazon Redshift and in-house data lake, using Apache data streaming projects to create a unified and flexible solution.
-
NVIDIA Introduces Metropolis Microservices for Jetson to Run AI Apps at the Edge
NVIDIA has expanded its Nvidia Metropolis Microservices Cloud-based AI solution to run on the NVIDIA Jetson IoT embedded platform, including support for video streaming and AI-based perception.
-
Real-Time Data Streaming Capabilities with AppSync Integration in Amazon EventBridge Event Bus
AWS recently announced that Amazon EventBridge Event Bus supports AWS AppSync as an Event Bus's target, enabling developers to stream real-time updates such as sports scores from their applications to frontend applications, including mobile and desktop.
-
DoorDash Develops New Sessionization Platform with Flink to Improve Notification Delivery Timeliness
DoorDash has significantly enhanced its user engagement by leveraging Apache Flink for real-time session detection and notification delivery. This move marks a substantial advancement in user interaction and cart conversion rates.
-
Expedia Uses WebSockets and Kafka to Query Near Real-Time Streaming Data
Expedia created a solution to support querying the clickstream data from their platform in near-real time to enable their product and engineering teams to explore live data while working on new and enhancing existing data-driven functional use cases. The team used a combination of WebSockets, Apache Kafka, and PostgreSQL to allow streaming query results continuously to users’ browsers.
-
How HubSpot Uses Apache Kafka Swimlanes for Timely Processing of Workflow Actions
HubSpot adopted routing messages over multiple Kafka topics (called swimlanes) for the same producer to avoid the build-up in the consumer group lag and prioritize the processing of real-time traffic. Using a combination of automatic and manual detection of traffic spikes, the company ensures the majority of customers’ workflows execute without delays.
-
Goldsky’s Streaming-First Architecture for Blockchain Data with Flink, Redpanda and Kubernetes
Goldsky created a platform for the real-time processing of blockchain data. The platform allows clients to extract data from blockchains into their own databases to support product features, but without running the data pipeline infrastructure. The event-driven architecture (EDA) of Goldsky leverages Apache Flink, Redpanda, Kubernetes, and cloud provider services.
-
Amazon MSK Replicator: Active-Passive and Active-Active Clusters for Apache Kafka Service
AWS has recently announced MSK Replicator, a new option for cross-region and same-region streaming data replication. The new feature of the Amazon Managed Streaming for Apache Kafka service provides automatic asynchronous replication across clusters, enhancing availability and ensuring business continuity.
-
Chronon - Airbnb’s End-to-End Feature Platform at QCon SF 2023
At QConSF, Airbnb staff software engineer Nikhil Simha presented Chronon, Airbnb's solution to address the challenges of managing and serving the vast number of features used in machine learning models. The platform focuses on four key areas: core APIs, training data generation, feature serving, and feature observability.
-
Confluent Announces Apache Flink on Confluent Cloud in Open Preview
Confluent recently announced the open preview of Apache Flink on Confluent Cloud as a fully-managed service for stream processing. The company claims that the managed service will make it easier for companies to filter, join, and enrich data streams with Flink.
-
Digital Ocean Launches its Managed Kafka Service
Digital Ocean enters the arena of fully-managed Kafka services with its new offering aimed at simplifying management and maintenance of the popular event streaming platform. Digital Ocean Kafka targets startups and SMBs by offering them an all-inclusive, flat-rate pricing model.
-
Allegro Uses Control Theory for Workload Balancing in its Apache Kafka PubSub Platform
Allegro, the largest eCommerce platform in Poland, implemented dynamic workload balancing in Hermes, its open-source publish-subscribe message broker, built on top of Apache Kafka. The new workload balancing algorithm achieves more uniform resource utilization and lower infrastructure costs.
-
Running Apache Flink Applications on AWS KDA: Lessons Learnt at Deliveroo
Deliveroo introduced Apache Flink into its technology stack for enriching and merging events consumed from Apache Kafka or Kinesis Streams. The company opted to use AWS Kinesis Data Analytics (KDA) service to manage Apache Flink clusters on AWS and shared its experiences from running Flink applications on KDA.