Recently, the Internal Developer Experience team at Discord talked about how they migrated all backend and infrastructure development to a Linux-based Cloud Development Environment. Through collaboration with Coder, developers were shifted from using MacBooks to working on machines in a remote development environment.
Denbeigh Stevens, senior software engineer at Discord elaborated the journey in a blog post. Discord engineering team primarily operates remotely. Before shifting to remote development machines, the engineering team could set up Discord on both Mac and Ubuntu systems. Equipped with MacBooks, the developers initially used Homebrew for laptop provisioning, where they faced challenges with brew upgrades disrupting work.
Before long, the Developer Experience team recognized the lack of containerization in the development process. As the organization scaled, the team realized the need for a single Linux-based development environment. This led them to look into Cloud Developer Environments (CDEs) and start considering Coder.
Since a majority of Discord's engineers utilize VSCode, the remote development extension for VS Code was beneficial in this scenario. Discord's approach included keeping the /home directory intact through restarts. This feature allowed the developers to continue their work seamlessly, storing repositories, settings, personal tools, and workspace customizations. They could also update templates and images without needing to rebuild their entire workspace.
Talking about challenges, Stevens mentioned that the delay caused by SSH and transferring large HTML and JavaScript files across the network slowed down the save and rebuild process. So some of the engineers preferred to handle frontend tasks on their local laptops and backend tasks on remote machines.
Source: How Discord Moved Engineering to Cloud Development Environments
The beginning of the migration involved developing the development container, which included installing necessary system dependencies, configuring user accounts, setting permissions, and installing any previously required software. To support this process, the Discord team conducted interviews, gathered early feedback, and also tested the environment firsthand by using it themselves.
To encourage adoption across Discord, the developer experience team sought out "champions" from different departments. They collaborated with these champions to identify and resolve issues, benchmark build tools, and run network load tests to maintain the development workflows.
This seems to align with an intriguing Hacker News discussion related to developer experience, where one of the members codingdave mentioned, "Dev experience is about people, not tech. You will use tech to improve it, but your devs are your customers - talk to them."
The migration was also set with a firm cutover date, simultaneously encouraging the staff to switch to Coder.
After the migration, engineers faced issues involving network latency and frequent disconnections. While predicting the most severe outcomes was difficult, the DevEx team utilized Coder's early satellite feature to set up Kubernetes clusters in various regions, aiming to reduce latency issues.
Stevens also pondered on the importance of improving communication and documentation. Despite announcing this upcoming change in all-hands meetings, signaling it early, and conducting a thorough beta testing phase, he noted there was room for enhancement in these areas.