Agile at Red Hat

A story of how Red Hat has embraced agile as told through the Red Hat Mobile team’s recent conversion project

Moving from a traditional waterfall approach of predefined roles and stages to a model of team ownership is a jarring experience for most. However, it's an experience that more teams and products will encounter as the move towards cloud computing gains momentum. Cloud deployments, embodied by the SaaS (software as a service) model for building and consuming services, are becoming more affordable and more desirable from a customer's perspective, where a need to respond to change rapidly is a core business value. Red Hat is undergoing that transition at present, as customers and consumers move from a model of self managed on premise solutions to the power and flexibility of the cloud.

Natural resistance to losing technical ownership, combined with a faster pace of development along with incremental deadlines approaching, is one of the main reasons why agile conversions fail in a company. In 2014, Red Hat acquired FeedHenry, a SaaS-based mobile platform which joined an experienced SaaS development team with an experienced waterfall cadence quality team. This article is a story of the successful conversion journey we undertook, and the learning curves we faced as we navigated the unique new home that Red Hat afforded us. It’s also charting the journey of agile as a whole in Red Hat, as this story is being successfully replicated across the product suite that Red Hat offers. From the operating system to the application server and several layers in between, Red Hat is a truly agile organisation and performing this transition the open source way.

An acquisition

FeedHenry was a startup based in Waterford, Ireland, that emerged as a very mature SaaS product in the mobile domain. In the autumn of 2014 they were acquired by Red Hat. The ex-FeedHenry team was well used to working in a very fast manner and would have been well versed in multiple releases a week, turning around features quickly for customers and operating in that frantic start up mentality where every hour of every day matters.

The acquisition by Red Hat was a dream for the staff and the initial months of bedding in was very much business as usual for the ex-FeedHenry staff as they still had customer requests and a roadmap to adhere to. Meanwhile, Red Hat began to put in place the fundamental building blocks that it has built its reputation on. Several support teams were introduced to assist the FeedHenry team in evolving what the FeedHenry product was and help transition it into the Red Hat Mobile Application Platform (RHMAP). This included the introduction of dedicated security specialists -- who protect Red Hat’s customers from vulnerabilities --, a program manager -- who acts as a coordinator and integrator --, an internationalisation team -- who translated our platform to Japanese, and the merging of the existing Red Hat open source mobile (Aerogear project) team --who led the charge with open source mobile tools --  into the wider RHMAP team.

The biggest addition was the access to a more rigorous Quality Engineering (QE)  team -- who looked after all things quality from the tooling to the testing suites to the release sign off. QE in Red Hat is different than the regular Quality Assurance (QA) work that other companies might have. QE are much more focused on the creation and maintenance of testing suites, the automation of our test execution and the overall quality heartbeat of a product. QA, on the other hand, is typically a softer validation, often a case of following a set of steps in order to verify that the bug you fixed is indeed resolved, or a set of steps to showcase the new functionality added. The real value, from a quality perspective, is in the QE side, which gives the most value and protection. That is not to diminish the importance of a QA pass on functionality. FeedHenry now had a much more stringent and rigorous quality team, which could only benefit our product and customers.

This was a huge learning opportunity for both sides. The ex-FeedHenry staff brought a lot of domain knowledge about how a SaaS platform should run. This included deep domain knowledge in mobile, the Node.js language and ecosystem and concepts, such as the critical role that devops and a Networks Operation Center (NOC) which would have been somewhat alien to the typical self managed or “on premise” solutions that Red Hat would have historically had. That led to the first year being an information exchange, with the support teams skilling up in the world of mobile, from the language to the processes and helping to put a framework of success in place. On the other side, you had the FeedHenry team adjusting to life in an open source company with huge support networks available to them. A change here was relinquishing control of areas in which they, as a startup, would have had to upskill in and manage. Now, in Red Hat, dedicated teams were able to take ownership and allow the FeedHenry team focus on their strengths. Getting the chance to open source and share their domain knowledge was huge for the staff. They embraced it but equally struggled with leaving the start up mentality behind. It was a classic conversion problem which Red Hat is well versed in, and the support the FeedHenry team received was astounding.

The Transition to Agile

We get the question asked to us a lot of the time, how did you transition to agile? The answer is, slowly! Unfortunately no magic button exists nor does a pre-defined map of the transition. The journey itself becomes clearer as you navigate each hurdle but the beginnings of the journey are the most vital. That is the point in time when you know the transition is needed and more importantly, it’s a reference point to look back on and see how far you have come.

For Red Hat Mobile (RHMAP), 12 months after the acquisition the honeymoon period was well and truly over. The support teams in Red Hat, primarily driven by the QE team, had become well acquainted with the FeedHenry platform and had released small versions of RHMAP to get comfortable with the process. The first major release under Red Hat’s guidance, our fabled 3.6 release, was due to stay in a QE cycle for 20 days. It took over three months to get that release into shape. Coding was continuous during this cycle, as requirements were changing and the team as a whole found it hard to say no to their loyal customers. QE were finding problems and as they were working downstream from the development waterfall, the cost of change was high and the development fixes were slow due to the passing of time and context shift needed to switch back in and address a bug in a piece of functionality that was worked on several weeks back. The team simply had enough. We now had our reference point and the right motivation to enact positive change. We understood we had to release quickly and we understood the need for rigorous QE cycles, as Red Hat prides itself on the quality of its products. Fundamentally between engineering and QE, we understand how we could bridge the gap in expectations, yet we struggled to move ourselves in an Agile direction. They needed help and thankfully it was an opportunistic timing for Griffin personally, as he joined the team to help guide the transition. Albeit, the expectations were well set that the full conversion would be a 12 month project.

We took the conversion from two angles: the team and the management.

The Team Conversion

The first six weeks were spent onboarding with the product and hearing the war stories of the most recent release that caused so much turmoil and heartache. We got a lot of background, recommendations and complaints as everybody had a valid point and Red Hat is built on a meritocracy, where every voice is equal and the best idea wins out!  I made some notes and simple observations as an outsider looking in, and in the typical spirit of any team or product starting on an agile journey, I made slight alterations and began with a single team (out of seven teams). The team was already used to following a very loose interpretation of Scrum, so we stuck with it but we started enforcing the Scrum ceremonies and making people aware of the value they can bring. We introduced a definition of done and helped bridge the gap in expectations between QE and Engineering with respect to the Quality Assurance (QA) work. That was the pivotal moment where the teams realised that work cannot simply be thrown over the fence. The QE team brought a huge value to the team, teaching others about quality processes and got to invest a huge amount of time in building regression suites and tooling to help protect the product. The time consuming and often boring work of performing a manual QA pass to ensure that the work performed actually meets the requirement set out was distributed among the team. The burden of quality is owned by everyone and that really was the lightbulb moment in our journey. Over the following weeks and months we saw huge improvements in feature velocity and overall happiness of the team, which helped get the crucial managerial buy in. That allowed us to accelerate our plans, and more teams came on board until all seven of our combined QE and engineering teams were following an agile methodology.

The Manager Conversion

We often talk about peopleware being the most difficult part of any transformation. Hardware and software are easy, but it’s peopleware you need to watch out for. This holds more true when dealing with managers and more senior figures, many of whom have carved their career based on decisions and processes which you may be about to undo with an agile conversion! With that in mind, the move to an agile methodology needs to be accepted at all levels; management cannot simply mandate this change, nor can it be a bottom up approach. When selling agile to both managers and engineers, you have to be transparent and open in your intentions. We could explain all the benefits, but equally we have to expose the fact that there will be a lot of discomfort on this journey. We had to encourage the team to fail (within reason!!) and learn from it. In essence, we had to reset expectations and had to allow the team and management find their equilibrium. Thankfully, at Red Hat accountability is one of our core values and the team quickly grasped that they could steer a sprint towards the rocks or towards safety based on their own decisions. Our path to self organising the teams was made much easier by this virtue and by the style of management that Red Hat embodies. We are people managers, and in Red Hat, that holds a lot of weight and expectations. We get a lot of natural 1:1 time with the team and opportunities to guide, coach and mentor as part of our day-to-day lives. From an agile coach’s perspective, it’s a dream job. We get to see real positive change and work closely with both teams and individuals, and our management style and interactions radiate outwards to peer managers, and upwards towards executive level. We had the full support of management to take on this journey and our successes as a SaaS agile based team allowed the radiation of our thoughts and processes to a much wider Red Hat wide initiative. Small changes and small steps are turning into a much bigger agile revolution in Red Hat, one which we are proud to have played a small part in.

Agile Accelerated

12 months after introducing agile, all of our teams within Red Hat Mobile were now following a flavour of an agile methodology, with most in a modified scrum framework and some running as kanban teams. This did not mean the transition was complete; adjustments were required. After the initial period of implementing agile, we carried out a much more rigorous retrospective of our journey to date, as anniversaries tend to be a good pivot point for such a task. Our findings led us to question some of our approaches, and we focused our efforts on improving a number of areas. We implemented a variant of scaled scrum to ensure alignment, as the teams would feed enhancements, feature requests and bug fixes into our major releases. This gave us a much better control over our releases. Sprints boundaries were tweaked to finish on a Wednesday instead of the traditional Friday. We had noticed that many team members took leave on Fridays, especially when we hit a bank holiday, and this badly affected the scrum events associated with a sprint end. Three week sprints were adopted as we could more easily account for outside influences, unexpected staff absence, external meetings and effectively become more reactive to our customers’ needs. On a weekly basis, we saw an increase in development velocity.

Now that the transition to agile had reached a level of maturity in all seven teams, we looked to trial it in small dynamic teams. We moved slowly away from the concept of static teams, which would have had minimal staff rotation because of deep rooted skills and requirements specific to each team. Over time we broke information and skill silos and cross skilled the team to allow us to take a more holistic view on team composition. We now started to assemble the right skills for the right job, and notwithstanding the headaches of timing issues for when to start and stop efforts, we started to see the real potential of the team. Features were shipped faster, the team was more engaging and more reactive to customers than ever. We hit the horizon goals and our small tweaks, spread over a long time, helped the team transition from a variation of agile to actually being agile in both thought and action. The journey is never quite never complete; our horizon goal has rightly shifted again and it should always challenge us to adopt and evolve.

Lessons Learned

To say we learned a lot was an understatement, so we will try and focus what we learned on three areas.

Breaking negative connotations with the term agile. A lot of the pain we encountered was caused by negative past experiences where staff members worked in companies that tried an agile conversion and failed. They tried to move too fast and after a one-day training course expected the team to transform and perform. Agile became a stick to beat them with and my initial few weeks were spent holding 1:1 conversations and retrospectives with each team member to try and understand their background and negativity towards wanting to work in an agile manner.

People are the heart of what you do. You can have all the technology and processes in the world, but until you have the right people with the right attitude, you are at nothing. For the transition to agile you need people who are self driven, innovative, open minded, capable of being a team player and most importantly, have the capacity to communicate. In Red Hat we put a large emphasis on people; it’s our core philosophy and something we pride ourselves on. As an agile coach or facilitator, it is imperative that I can communicate on an equal level with members of all inter connected teams. What this meant for the agile transition was that we had to be capable of giving and receiving feedback, both positive and negative. We needed the ability to be able to rationally explain and support our decisions, and accept when we had got them wrong. We failed as facilitators several times, and it’s the reaction when that happens that is the most important for the health of the team and for the greater good of the project’s transition. Setting the right tone, showing respect, and never mandating and listening should be the mantra you adopt.

The third area where we learned is the importance of a support group. We were fortunate to stumble across an agile coach’s group within Red Hat. This allowed a safe place to talk, share experiences, share problems and share my own knowledge. The value of a peer group was huge for me and gave me additional material and insights that I could bring to the team. As FeedHenry were 12 months into their Red Hat journey, it allowed us to deal with some of the nuances that working and living in an open source world brings: the challenges of community and product expectations, the mentality of open source developers, the interconnectedness of the Red Hat ecosystem. We could not have achieved that without having a strong and open peer group to connect with. I would encourage any agile advocate to find a local meetup (or start one!) or go to great conferences like Agile Cambridge where they can meet and connect with peers and bring something new back to their team. The agile conversion is as much a conversion and journey for you as a person as it is for the team or product you are bringing along with you. Finding like minded products, companies and teams that are similar in style will really help you and your team grow into the task at hand.

A word on culture and the remote world we live in

Culture is a variable that more and more companies have to account for when looking at an Agile journey. Different cultures can and will dictate interactions as well as behaviours within the team. People from different backgrounds or cultures will have different perspectives, and these perspectives can often intensify when the team members are not co-located. Red Hat is based on a remote culture and has grown the company with hugely talented individuals scattered around the globe. Half of our team are co-located with the other half distributed on UTC friendly timezones. It is imperative that we keep a good communication channel open for our remote team members and involve them at every opportunity as if they were co-located in the office with us. We do this by having weekly one-on-one conference meetings with all remote team members. We think it is very important to connect with people on a personal level, as too many companies and managers view their team members as a resource. A resource to us is a laptop, or a cloud instance. Our team is made up of talented individuals who bring a unique personality and skillset combination to the table. You have to leverage that to build successful teams. You have to connect with your team and understand their background, and more importantly, where they are going career wise. We hold all of our meetings over remote communication mediums, even if 99% of the team are in the office, we resist the urge to hold a physical standup and dial one person in. Instead, we all take our calls at our desk and adopt a remote mentality to ensure parity within the team.

We put a lot of time every single week into this and we see the rewards in the success of our teams and our products. A big help for us, in Red Hat, is the fact we have a unique culture which almost trumps individual cultures. It has a huge focus on community and the values of open source, which align really well with the values and principles enshrined in the agile manifesto. I think bringing or forming an overarching culture to a team will help bridge any individual cultural issues. We welcome the diversity of our team and the cultural diversity helps us grow and learn as a team.

About the Authors

Dr. Leigh Griffin is an engineering manager in Red Hat Mobile based out of Waterford, Ireland. Griffin has an extensive software background attained over the last decade in industry, academia and research. In recent years a specialisation in agile transformation has seen him employ this experience on large conversion projects.

Brendan O’Farrell is an engineering manager in Red Hat Mobile based out of Waterford, Ireland. O’Farrell has over 25 years of business and process management in an array of industries. Having converted to Software Engineering later in life, he is now helping Red Hat move more teams towards agile and sharing his vast knowledge on the subject.