On Nov 17th, the 2nd day of QConSF 2015, Bison Liao the Director of Tencent Social Network Group gave a presentation "The Philosophy of Mass Services at Tencent" on the China Tech Day track.
This article is an overview of Tencent’s technology evolution through the past 17 years since its birth as IM software.
From 0 to 100 million users
On March 5th of 2010, the number of simultaneous users of QQ IM exceeded 100 million. There have been several talks and articles revealing the changes in the backend structures behind this growth, describing three major structure refactorings at various stages.
The first major refactoring stepped in at a scale of a million simultaneous users. The initial backend was a simple structure that consisted of a storage server and an access server, with an additional server to provide long connections for users without direct connection to the access server. After the first refactoring, a distributed cluster was applied over the single server model, adding a server to sync user status among access servers. When the new product feature QQ Groups was published, a similar infrastructure was built for this new feature. The main challenges during this period was high availability and 24/7 continuous service.
The second major refactoring occurred during 2005, when QQ reached ten million simultaneous users. As of this time, the single status syncing server could no longer serve all the users, so a distributed cluster structure was applied to the user status syncing service as well. By the time all servers serving QQ and QQ Groups were under the same IDC, and more work was put into implementing geographic failover clusters. There was also much work put into orchestration automation and stage deployment.
The next major refactoring happened during 2010, when QQ reached a hundred million simultaneous users. Mobile internet was growing, and the world of the internet became faster than ever. Development of new features needed to be faster, so there was more requirement on flexibility. The entire system was decoupled down to two major systems, the storage system and the communications system, and each was further decoupled to various services. This new structure became the basis of the latter Tencent Cloud.
Philosophy of Tencent Development
Tencent engineers have summarized the philosophy behind their development model as follows:
- Minimalism - The best example for this is WeChat. WeChat 1.0 provided nothing other than the core chat feature. Feature updates were frequent (about 1 major update per week with various small backend updates everyday) but each update only does one new thing. Today's WeChat is one of the largest online platforms for content providers.
- Stage deployment - Stage deployment within Tencent is more than Dev-Staging-Production process for software quality control, but also serves bucket testing purposes to reduce risk on the product design level. A normal stage deployment follows a sequence of 5 stages on the production environment: internal employees -> core users -> 10% of active users -> 50% of active users -> all users.
- Decoupling - Tencent does modular design at a rather fine level. Component reuse across different businesses is common.
- Service degradation - Connections to major interfaces are prioritized, and connections to secondary interfaces are allowed to be degraded when resources are low. By using smart design on the front-end, users could still use core features without much problem during service degradation.
Summary
Today's Tencent has strong products - QQ, Groups, Space, WeChat, Games, Cloud, etc. Supporting millions even billions of users while maintaining a good user experience; things that would not be possible without strong technical teams. There are definitely good things we could learn from them, and China Tech Day is a good chance to talk to them!