Managing 238M Memberships at Netflix

Surabhi Diwan, a Senior Software Engineer at Netflix, presented at QCon San Francisco 2023 on Managing 238M Memberships at Netflix. In her talk, she shared how the Netflix membership team does distributed systems: the architecture bets, technology choices, and operational semantics that serve the needs of Netflix’s ever-growing member base.

She started with some of the pricing and technology choices Netflix made probably a decade ago that predate her tenure at Netflix. She then moved on to a use case study for member history, which is the second persistent store that helps us answer questions about the microsecond granularity of any and every single change made to anybody's subscription.

"I'm sure most of you are Netflix members. If you're not, I will show you exactly how to sign up as we get into the details of this. Finally, I will try to answer the question: What does the subscription ecosystem evolution look like? It has 238 million subscribers. Really, what is the journey like? What does that look like if you have to add another 5 million subscribers? If you have to add another 100, what does that look like?"

Membership Engineering

The team's primary focus with memberships lies in the critical pathways of signups and streaming at Netflix. They manage a suite of various microservices, totaling a dozen within the membership domain. With a commitment to four nines availability, their mid-tier services ensure uninterrupted user access, directly impacting signup processes and streaming experiences.

These services handle substantial traffic, scaling up to millions of requests per second depending on the specific use case, such as subscriptions or pricing. The team discusses their technical decisions, leveraging off-the-shelf technology to achieve scalability and reliability. They serve as the authoritative source for global membership counts, facilitating downstream analytics both within and outside Netflix. In essence, they tackle complex distributed challenges at scale with precision.

Additionally, the team oversees the catalog of Netflix plans and pricing, which, despite its simplicity - basic, standard, premium - plays a crucial role in user experience. They prioritize accuracy in subscription details, ensuring correct plan selection, billing country, and payment status to maintain service quality and member satisfaction.

Managing the entire membership lifecycle, they facilitate signups, partner channel integrations, plan changes, renewals, and account closures. Their responsibilities include handling payment issues, including account holds and cancellations, and ensuring compliance with data privacy regulations by appropriately managing customer data throughout the membership journey.

When Do Members Use Our Flows?

I think that was a broad establishment of what my team does. As end users, I think you would care if you were to go on the Netflix app right now. These are some of the places.

The flow highlights critical user interactions, such as the start membership and play buttons, which trigger backend processes managed by the Netflix Membership engineering team. A flowchart illustrates the various services ensuring a seamless membership experience before users can begin streaming. The play button directly interacts with membership systems to determine service quality based on the user's plan. This interaction generates the highest traffic volume due to the billions of streams initiated daily. Additionally, user actions on the account page, such as plan changes or managing additional members, are facilitated by membership services. Partner signups, exemplified by Xfinity activations, are also orchestrated by the team's backend services.

How Do We Make It Happen?

I think this is the meat of the puzzle: establishing what we do. This is really how we do it. It is a little hard to unpack.

The team manages the membership plan and pricing catalog, storing and managing plans globally, with variations across different regions. This service also handles rule management based on location-specific offerings. They utilize two CockroachDB databases to support plan pricing and code redemptions, particularly during peak gifting seasons. The member pricing service facilitates member actions, such as plan changes and adding extra members.

Partner interactions are managed by a dedicated microservice that handles bundle activations and signups, including integration with platforms like Apple's App Store for subscription signups. Membership data is stored in Cassandra databases, supporting subscription services and history tracking for over 238 million active memberships.

The team's focus extends beyond current members, including former members and rejoin experiences. They manage states with services like member states and member holds, ensuring smooth operations and reconciliation across databases using tools like Casspactor and Apache Spark for big data processing. This data, such as messaging and analytics, is crucial for downstream consumers to derive insights on signups and revenue projections.

Signup Journey

Once users begin their journey on Netflix, they encounter plan selection options powered by membership systems, which handle millions of requests per second. Rendering this page correctly is crucial due to geographical variations in currency, pricing, and available plans. The team manages these rules, with green boxes representing membership services' responsibilities and white boxes representing collaboration with sister teams.

The process starts with plan selection through growth engineering apps. The chosen plan is queried from the membership plan and pricing service, backed by CockroachDB for plan pricing details. After confirmation, hitting "start membership" triggers actions in the membership state and history services, updating persistent records with relevant information like plan, price tier, and country. Events are emitted to signal the membership's activation, triggering messaging pipelines for welcome emails and informing downstream teams for analytics. Despite the simplified explanation, this process operates at a large scale within a distributed system, requiring robust error handling and reconciliation.

Membership Team Tech Footprint

Netflix operates on a distributed systems architecture optimized for high read requests per second (RPS). With over 12 microservices, they utilize gRPC for communication at the HTTP layer. Their primary coding languages are Java, and they are transitioning to Kotlin for app rewrites, all brought together with Spring Boot. Kafka plays a significant role in message passing and interfacing with other teams, such as messaging and downstream analytics. Additionally, Netflix employs Spark and Flink for offline reconciliation tasks on their big data, which we'll explore in more detail later.

Technology Choices for Operations and Monitoring

Beyond coding and deploying to production, the Netflix Membership engineering team takes on-call responsibilities, addressing critical failures promptly. They employ lightweight transactions and try to ensure data consistency in our online systems by utilizing tools like Cassandra. Reconciliation jobs, powered by Spark and Kafka, ensure alignment between systems of record within the membership, such as subscriptions and member history databases. This accuracy extends to external systems, maintaining a consistent state across the ecosystem. Data alerts and repair jobs monitor and rectify discrepancies, ensuring every record reflects the latest information.

In observability, extensive logging, dashboards, and distributed tracing facilitate rapid error detection and resolution, which is crucial in Netflix's vast microservices landscape. Production alerts track operational metrics, guaranteeing optimal service levels. Operational data also fuels machine learning models for enhanced anomaly detection and automated issue resolution, maintaining uninterrupted streaming experiences for our members.

Use Case Studies

So far, we have established the Netflix Membership engineering team's position, architecture, and core operational flows. It's vital to delve deeper into potential improvements and future readiness. Comparing system design to chess, mastering it requires understanding rules and strategies and analyzing past moves for improvement.

We will reflect on Diwan’s five-year tenure; the team examined past architectural decisions, mainly focusing on Netflix's pricing, technology choices, and the importance of cataloging system modifications. In addition, we explore their journey in scaling, from initial requirements to continuous evolution and improvement.

Learning from the Past - Netflix Pricing Tech Choices

A decade ago, Netflix's pricing architecture was simplistic, managing only a few plans and basic functionality. Initially, a lightweight, in-memory library served these needs efficiently. However, as Netflix expanded globally and diversified its offerings, this library grew in scope and complexity, becoming vital across multiple applications. Over time, operational challenges emerged due to its size and dependencies, necessitating a transition to a more robust architecture.

The new plan pricing architecture utilizes CockroachDB for persistence and a gRPC service to manage traffic. Despite the simplified design, migrating from the legacy library was a multi-year endeavor involving numerous engineering teams and applications. This highlights the importance of future-proofing architectural decisions and promptly addressing tech debt to avoid costly issues.

While the new architecture is the primary solution, remnants of the old library still linger, requiring ongoing migration efforts. This underscores the need to consider long-term implications and proactively address legacy systems during technology transitions.

Member History

The study on member history dives deep into its evolution and pivotal role within Netflix's architecture. Initially, member history was tracked through application-level events, but the need for granular data persisted. As Netflix expanded globally, the complexity and criticality of member data grew, necessitating a more robust solution.

The new architecture employs a change data capture pattern, recording direct delta changes on operational data sources. This append-only log system, backed by a Cassandra database, offers comprehensive tracking of membership events. By centralizing event emissions through member history streams, Netflix gained improved observability and the ability to reconcile data across systems.

The benefits of this architecture are manifold. It enables detailed debugging, replaying of events, and seamless reconciliation in case of data corruption. Moreover, member history enriches customer service analysis and feeds into downstream analytics, messaging, and membership data systems.

Despite the multi-year journey to implement this architecture, the dividends have been substantial, underscoring the importance of investing in architectural innovations for long-term success.

Preparing for the Future - Member Subscriptions Ecosystem Evolution

As we enter the final stretch, let's delve into the evolution of the subscription ecosystem. Initially, we made basic architectural choices and relied on off-the-shelf components like gRPC services and Cassandra databases for scalability. However, as our user base grew, we encountered challenges reconciling data and ensuring fault tolerance.

To address these issues, we implemented a Spark Casspactor to manage backups and reconcile data in Hive tables, enabling better auditing and self-healing. While this improved debuggability and removed single points of failure, scalability remained a concern. To mitigate this, we're considering caching with EVCache for faster lookups, albeit with some trade-offs in consistency.

The key lesson here is that no system can infinitely scale, and it's crucial to continually invest in innovation and architectural evolution to stay ahead of system limitations and avoid unexpected outages.

Conclusion

The pivotal lesson from Netflix's pricing decision is the importance of future-proofing its technology choices and being proactive in adapting or pivoting when necessary. Similarly, the member history case underscores the potential payoff of bold architectural investments. It's essential to have the courage to pursue significant innovations.

The evolution of member subscriptions is a continuous process with no definitive endpoint. This ongoing challenge reminds Diwan of a famous quote in computer science:

There are only two hard things in computer science: cache invalidation and naming.

While Diwan focuses on caching, she recommends watching "The Queen's Gambit" for a captivating story about pursuing excellence.