This article is part of the “DevOps War Stories” series. Each month we hear what DevOps brings to a different organisation, we learn what worked and what didn’t, and chart the challenges faced during adoption.
Learning DevOps from within
Introduction
Prezi’s philosophy of running each product team like a “mini-startup” - as decentralized as possible - means every team needs to be self-sufficient. This is only possible using the classical devops recipe of mixing software developers with engineers experienced in operating large-scale systems. As the executive team at Prezi realized this, the push for company-wide devops was born. With this initiative, the question of how employees would develop the skillset necessary to implement devops became especially important.
Prezi has long had cross-functional teams consisting of designers, QA engineers, developers, user experience researchers and product managers. Until recently, however, there was only one “devops team”. Today, this team is still responsible for the availability of most mission-critical services like core databases and HTTP load-balancers. What changed is that individual product teams have assumed ownership of the infrastructure running their products or features. Many of these teams did not include anyone with devops experience, so it became a priority for them to learn the basics.
The traditional method of learning a technology stack is to enroll in courses typically organized by vendors, often to earn a certification. While this tried-and-true path has many benefits, it’s a heavy-handed approach, requiring a significant investment of money and time from both the company and the individual employee. Those who are familiar with much of the material in an introductory course but are not yet ready for the next level will either be bored or confused. Finally vendor courses often woefully neglect the “big picture” focusing primarily on the problems solved by the company’s product line, not the problems faced by those enrolled. Because of these drawbacks, engineers are much better off learning from their colleagues if they can.
Fortunately, Prezi started experimenting with different methods of sharing information internally long before every team turned devops, so when we really needed effective ways of sharing the knowledge we had within the company, we had them. The purpose of this article is to share these techniques.
Bootcamp
Learning from teammates is easy if you know them well, but what about new hires? Prezi struggled with getting new engineers up to speed on the codebase and company values. The solution was to scale down Facebook’s impressive 6-week training marathon and create our own two-week bootcamp. New hires spend up to a day with each of the teams that make up the organization. This introduction week is designed to impart several different kinds of knowledge.
First of all, spending some time with every team at Prezi is a great way to get to know one’s new colleagues. It’s much easier to remember names after a short conversation or crash course on an interesting topic than after the usual whirlwind tour with a thousand handshakes. In addition to pairing names with faces, graduates of bootcamp will also have a good idea of who they can approach with a specific problem.
Second, bootcamp gives new devops engineers the bird’s eye perspective of Prezi’s infrastructure and code. Typically, after a short presentation, participants will be given a small but reasonably challenging task. Examples include writing a chef recipe for configuring an apache instance and creating a dashboard for the number of prezis exported for use with Prezi Desktop using data from hadoop. These mini-projects are carefully chosen to be doable in the time available, but also useful in their own right, not just “busywork”. They also typically require talking to two or three experienced engineers to get advice on how to solve the problem at hand.
Finally, bootcamp introduces new employees to what working in a startup environment is like. At Prezi, there are teams, but no departments. We don’t have too many layers of management, but we do have a user experience research team. All of these differences can be a bit disorienting for newcomers. Prezi’s iterative development style is probably the most difficult new concept for fresh engineers to integrate into their working habits, simply because it’s so pervasive. By iterative development, I mean following the build-measure-learn cycle. At Prezi, we try to apply this lean startup approach not just to major product decisions, but also to minor 15 minute tasks and everything in between. It is the responsibility of every engineer to avoid waste by building good-enough solutions which can be tested and, if necessary, scrapped without a major investment of time and energy. Prezi’s urgency in getting working systems out the door is the exact opposite of the quest for finding the perfect solution the first time around, so common in academia and large corporations. A recently joined team member remarked during bootcamp that hearing every group talking about rewriting this code, or rethinking that process during their presentation really drove home the iterative spirit. Now that he has graduated, not only does he know a lot more about how Prezi works, he also knows who could benefit from the knowledge he brought with him. The learning goes both ways.
Meetups
Meeting new people is a sure way of encountering new ideas. Meetups are an excellent way to find people from other organizations who face similar challenges. In addition to swapping war stories, talking to engineers outside the company can also serve as a reality check. For example, if we’re having a lot of trouble with a particular database, it’s important to know whether it’s a configuration problem on our end, or if the version of the product we’re using is giving others a hard time as well.
There are several technical meetups which Prezi hosts, often offering free pizza and beer to participants. It’s a classic win-win situation: for the organizers, having a suitable venue at their disposal free of charge is a big help. For Prezi, the arrangement has several advantages.
First, it makes it really convenient for employees to attend meetups related to their daily work. When one of my teammates gave a talk on Apache Kafka at the Budapest DevOps meetup, many of our colleagues were in the audience, since all they had to do was walk upstairs after work. The next day, everyone who attended was talking about how we should re-think the amount of hardware we allocate for services based on the last presentation that night.
Second, it’s a great way to foster employees’ technical interests. If I’m really into an area of research within computer science or a programming language, all I need to do is find enough people who share my interest. Since I don’t have to worry about the venue, the barrier to organizing a get-together is quite low. Even if the topic cannot be put to use in Prezi today, there may come a time when that piece of knowledge is exactly what’s needed to solve a problem our company faces.
Finally, hosting meetups brings talented engineers to Prezi. It’s obvious to anyone who walks through the entrance that Prezi is not the average software company. When these people come to a point in their life that they want to switch jobs, there’s a good chance they’ll send a copy of their resumé to Prezi.
Bouncing around an interesting idea that came up during a meetup within the team is an excellent way to start a technical discussion which may benefit every participant (even if the original idea turns out to be inapplicable to current problems).
Brown bag talks
Perhaps the oldest form of sharing knowledge, the lecture is alive and well within Prezi. Brown bag talks (named after the brown bags in which members of the audience bring sandwiches) are optional lunchtime lectures lasting 45 minutes to an hour. The topic of the brown bag talk can be anything at all.
We regularly invite guest speakers to hold such talks. Some of the more memorable ones include a crash-course on US-EU trade relations by the former Hungarian ambassador to the US, and a presentation on how the data services team at Spotify cope with their technical and organizational challenges.
Any employee is free to organize a brown bag talk. One of the best presentations was on the paxos algorithm and alternative approaches to data consistency. The presenter became an expert on the subject before he joined Prezi. Members of the audience walked away with a new understanding of what tradeoffs are possible. In our team, the engineers with operations backgrounds were frantically trying to remember if they accidently configured MongoDB with dangerous settings in light of what they had just learned about data consistency.
What sets these brown bag talks apart from a typical internal presentation is their informal nature. If someone in the audience realizes that this talk is not for them, they are free to leave. As a result, very few people bury themselves in their laptops or smartphones during such an event. The informal atmosphere also makes it easier for the speaker to deliver his point, since his message does not have to resonate with the whole company.
At the cost of requiring more preparation than a spontaneous conversation, such talks are an efficient way of sharing one’s expertise with colleagues.
Hackathons
The most authentic type of learning is the “on the job” kind. That doesn’t mean you have to figure everything out by yourself reading man pages, however. Solving unusual problems together is a great way for members of a group to learn from each other.
Prezi regularly organizes hackathons at its engineering headquarters in Budapest. Anyone interested is welcome to attend. The event begins with a brainstorming of novel ideas to implement. Mixed teams composed of both employees and guests form around popular ideas. These teams spend the next day and a half proving the feasibility of their idea through a “hack”.
Some of the teams work on prezi.com related projects, but this isn’t mandatory. A good counterexample is the group which wrote a web app listing who is currently in the office. The hack works by listing wireless devices connected to the company wifi network, a good mix of low-level unix hacking, python web application code, and some nifty javascript on the client side.
Hackathons are a good way to simulate real emergencies which occur in the life of a devops team. The parallels are numerous. In both situations, a heterogeneous team is dealing with a highly ambiguous problem. Time is a limiting factor, so the team needs fast solutions. Finally, the better each teammate can contribute with whatever is his or her strength, the more likely the team is to succeed. Aside from the technical aspects, the roles assumed by members of the team are also similar in both cases. Naturally, some members will take on coordinator roles, while others will prefer to “do instead of talk”. Needless to say, a successful team needs both kinds. Finding out which of these each team member does is vital preparation for the real devops disasters, like when a mysql master node dies leading to downtime and data loss.
In-house courses
The techniques described so far were all introduced as knowledge-sharing alternatives to vendor courses. Beyond a certain technical depth, however, these more casual approaches often don’t work, especially when the technical backgrounds of audience members are too diverse.
With their predictable schedules and moderate pace, courses give students an opportunity to thoroughly immerse themselves in a topic. One example of a course held at Prezi was a short introduction to haskell programming, held on four consecutive friday mornings. Every team in the company was represented. There were designers and QA engineers sitting next to developers with years of functional programming experience. By starting from basics and moving at a steady pace, an impressive amount of material was covered. Participants were encouraged to ask questions both from each other and the instructor (who was a colleague). As a result, there was always time for questions and suggestions on how the content of the course could be used in our daily work even if we had to deviate from the original lesson plan. This flexibility and open atmosphere made the course a success even though we use haskell in few production systems.
Structured in a similar way to the haskell course, a reading group was launched focusing on classical computer science articles. Each week, a member of the group presents one of the papers from the reading list. A discussion follows in which any member of the group is welcome to comment on or challenge the presenter’s interpretation of the publication.
Vendor courses are generally taught by experienced presenters using professional teaching materials. They are an efficient means of one-way information transmission from an expert to an audience, and there have been cases where it made sense for Prezi engineers to enroll. Prezi’s in-house courses are not viewed as direct competition to vendors’ offerings. Instead, the reason for organizing in-house courses stems from the realization that the one-way model is not the only means of communication. In fact, it may not even be the most important one. After all, the way a devops team solves problems is typically a much less structured process, often requiring one to defend his solution to a problem or ask the right questions in order to quickly find the root cause of problems. In addition to providing useful technical information, these in-house courses and reading groups definitely help sharpen the much-needed soft skills required to make a team operate smoothly.
Prezi being a presentation company, it’s particularly fitting that emphasis is placed not just on the technical development of employees, but also their ability to share ideas. By improving the latter, engineers will also become better at helping each other in the former.
Conclusion
I have learned an incredible amount from my colleagues since I started working as a devops engineer at Prezi. It’s also a great feeling that I have helped others in solving problems and, ultimately, in becoming better engineers. Most people want to work in an environment like Prezi’s where everyone is encouraged to share their ideas and any idea can be challenged. The alternative knowledge sharing techniques Prezi employs go a long way in creating such a setting. Facilitating the unobstructed flow of information within devops teams is especially important, since, given team members’ heterogeneous backgrounds, every member of the team likely knows something others could benefit from. The same can be said for the benefits of establishing dialog between teams. Regardless of whether or not your company enrolls employees in professional courses, chances are it would benefit from trying out some of Prezi’s alternatives!
About the Author
Peter Neumark is a devops guy at Prezi. He lives in Budapest, Hungary with his wife Anna and two small children. When not debugging python code or changing diapers, Peter likes to ride his bicycle.
