Chen Yang and Fan Zhang at DoorDash's Engineering team have recently developed an in-memory sessionization platform with Apache Flink that identifies user activities and user sessions in real time. The new system shown by the diagram below solved one of the many challenges DoorDash previously faced - identifying when a user had genuinely abandoned their cart versus merely browsing for more items or different merchants. This distinction was crucial for sending timely and relevant cart abandonment notifications. Traditional methods, such as batch processing, were resource-intensive and lagged, making real-time detection nearly impossible.
Real-time session detection data flow, designed by DoorDash's Engineering team.
The new sessionization platform is built with a large, stateful Flink job to monitor user sessions through mobile and web client events in memory. This approach allowed them to process user activity event stream without "having to load massive amounts of data from cold storage to memory before processing", therefore "avoiding huge computational costs and latency" as well as "detecting periods of inactivity in short time frames, [in order to] send notifications at more opportune moments." The new platform isn't without its challenges. Since Flink processes all events in memory in real time, it needs to keep all user session events until the session ends. Given DoorDash's current scale and the fact that each customer session is about an hour long, the local inflight state can be hundreds of gigabytes at any moment.
DoorDash's adoption of Apache Flink for real-time session detection and notification delivery significantly enhances user engagement and conversion rates. The new design has helped DoorDash significantly improve notification delivery timeliness and accuracy, resulting in a 40% higher send-to-open rate. The success has already led to a new series of experiments with the platform, including adding promotions to cart abandonment notifications and controlling the frequency of cart abandonment notifications.