InfoQ Homepage Event Stream Processing Content on InfoQ
-
Ahead of re:Invent, Amazon Updates AWS Lambda
A series of updates to AWS Lambda aim to improve how the function-as-a-service platform handles asynchronous workflows and processes data streams. These newly announced features arrived the week before the annual mega-conference, AWS re:Invent.
-
Jay Kreps: Events, Event Streams and Their Importance in a Digital Business
Organizations are moving more and more processes into software, Jay Kreps notes in a blog post, and adds that in an accompanying change businesses are increasingly defined in software – the core processes are specified and executed in software. To support this transition, he believes we have to move away from traditional databases into working with the concepts of events and events streams.
-
Delta – a Data Synchronization and Enrichment Platform by Netflix
Large systems often utilize numerous data stores. There is sometimes a need to keep some of these data stores in sync, and to enrich data in a store by calling external services. To address these needs, Netflix has created Delta, an eventual consistent, event-driven data synchronization and enrichment platform. In a blog post, the team behind Delta gives an overview of their design.
-
Oleg Zhurakousky on Spring Based Event-Driven Microservices
Spring Cloud Stream and Spring Cloud Function technologies support the implementation of event-driven microservices for diverse use cases, ranging from application integration to data streaming. Oleg Zhurakousky, project lead for Spring Cloud Stream and Spring Cloud Function, spoke at SpringOne Platform 2019 Conference about the recent developments and features in both of these projects.
-
Jagadish Venkatraman on LinkedIn's Journey to Samza 1.0
At the recent ApacheCon North America, Jagadish Venkatraman spoke about how LinkedIn developed Apache Samza 1.0 to handle stream processing at scale. He described LinkedIn's use cases involving trillions of events and petabytes of data, then highlighted the features added for the 1.0 release, including: stateful processing, high-level APIs, and a flexible deployment model.
-
Data Engineering in Badoo: Handling 20 Billion Events Per Day
Badoo is a dating social network that currently handles billions of events per day, explains Vladimir Kazanov, data platform engineering lead. At Skills Matter, he talked through some of the challenges of operating at this scale, and what tooling Badoo uses in order to process and report on this data.
-
Amazon Managed Kafka Aims to Simplify Kafka Streaming Setup and Use
Introduced as a public preview at AWS re:invent 2018, Amazon Managed Streaming for Kafka (MSK) is now generally available. Amazon MSK aims to make it easy to build and run streaming applications based on Kafka.
-
Experiences Going from Event-Driven to Event Sourcing: Fangel and Ingerslev at MicroCPH
At MicroCPH 2019, Thomas Bøgh Fangel and Emil Krog Ingerslev, both at Lunar Way, a fintech company, described how after building a monolithic Rails application they decided to migrate to an event-driven microservices architecture. During the migration they found some design issues and decided to move to event sourcing. In their presentation they discuss the problems and how they solved them.
-
Event Streams and Workflow Engines – Kafka and Zeebe
In a recent joint meeting in Amsterdam, Kai Waehner described features of Apache Kafka, a distributed streaming platform, and how it fits in an Event-Driven Architecture. Bernd Rücker described how workflow engines can handle complex business processes, and discussed how Zeebe, a new highly scalable workflow engine, can be used with Kafka.
-
In a Serverless World, We Still Need State
Today there is a large push in enterprise architecture towards serverless architecture. Jonas Bonér argues that although he strongly believes in the serverless movement, the programming model should not focus only on stateless functions. We must also have a focus on state, allowing us to benefit from the advantages of serverless while building distributed general-purpose applications.
-
Creating Events from Databases Using Change Data Capture: Gunnar Morling at MicroXchg Berlin
When you store data in a database, you often also want to put the same data in a cache and a search engine. The challenge is how to keep all data in sync without distributed transactions and dual writes. One way is to use a change data capture (CDC) tool that captures all changes made. In a presentation at MicroXchg Berlin, Gunnar Morling described Debezium, an implementation of CDC using Kafka.
-
The Importance of Event-First Thinking
For global businesses to meet today’s architectural challenges with constant change and extreme scale, we need to go back to the basic principles of system design. The common element in the problems we face is the notion of events driving both actions and reactions, Neil Avery writes in a series of blog posts describing why events are so important and the advantages of an event-first approach .
-
Microsoft Announces the General Availability of Azure Event Hubs for Apache Kafka
Recently, Microsoft announced the general availability of Azure Event Hubs for Apache Kafka. Customers will get the best of both worlds—the ecosystem and tools of Kafka, along with Azure’s security and global scale.
-
Event Sourcing to the Cloud at HomeAway
Adam Haines, Data Architect at HomeAway, recently spoke at the Data Architecture Summit 2018 Conference about how his team leverages event sourcing cloud design pattern to accelerate the big data initiatives in their organization.
-
William McKnight on Data Platforms and Creating a Modern Data Architecture
William McKnight gave a keynote presentation last week at Data Architecture Summit 2018 Conference on creating a modern data architecture using different data platforms.