Susheel Aroskar from Netflix's Engineering team spoke at the recent QCon New York 2018 Conference about Zuul Push, a scalable push notification service that asynchronously pushes data like personalized movie recommendations from cloud to devices.
The Netflix team uses Zuul Push to improve the user experience for their customers. A typical user on Netflix spends a significant amount of time picking a movie compared to watching a movie. This framework helps in providing better engagement, by generating customized recommendations in the cloud that can be shown to the user. Also, if there is a better recommendation generated, it lets the client know in real time.
Zuul Push is a high performance asynchronous service based on Apache Netty, a non-blocking I/O (NIO)-based network application framework. Zuul push can be used to scale to large number of persistent connections, and supports WebSocket and Server-Sent Events (SSE) protocols for push notifications. The technology handles more than 5.5 million connected clients at peak.
Aroskar discussed the architecture of Zuul Push solution, which includes the following components:
- Zuul Push Servers
- Push Registry
- Push Message Queue
- Push Library
- Message Processor
The replicated client registry makes it possible to scale the concurrent persistent connections and deliver push notifications. The push registry, which is based on Redis and Cassandra, provides the features needed for this type of workload, which includes low read latency, record expiry, sharding and replication. Data is stored using Dynomite, a generic Dynamo implementation for different key-value stores, which provides access to a Redis key-value backing-store with additional capabilities like auto-sharding, read/write quorum, and cross-region replication.
He also reviewed the design of the backend message routing infrastructure which uses Kafka server to decouple message senders from receivers. Message processing is responsible for queuing, routing, and delivering the messages. The solution uses the "Fire and Forget" approach to message delivery and there are different queues for different priorities of the messages. Netflix engineering also run multiple message processor instances in parallel to scale the message processing throughput.
Zuul Push supports custom authentication policies so you can authenticate using cookies, JSON Web Tokens (JWT) or some custom scheme. Aroskar discussed the operational aspects of Zuul Push and how they managed long lived stable connections. They use Amazon's Application Load Balancer (ALB) to manage the push cluster.
Zuul Push framework is based on Zuul which is Netflix’s API gateway. All of the Netflix HTTP API traffic passes through Zuul. Readers keen to learn more about the Zuul Push notifications framework can consult the project's website and the sample Java application which shows how to use WebSockets as well as SSE to enable push messaging for Zuul.
