How to Evolve and Scale Your DevOps Programs and Optimize Success

The appearance and ongoing development of DevOps is arguably the most important transformation to the way that IT teams work of the last decade. By integrating continual development and iterative feedback into the operations lifecycle, teams in many industries have achieved huge gains when it comes to agility and resilience.

Many have done so via applying a specific set of strategies designed to integrate the workflows of developers with those of operations staff, and often firms take a cautious, gradual approach to this transformation. They will implement a DevOps workflow for a new product, for instance, before rolling out the same approach to their entire organization.

As we’ve previously pointed out, this is a wise approach. However, it is accompanied by a small downside – that it can be difficult to translate small-scale managerial and technological structures into organization-wide structures.

Processes and workflows inevitably become more complex and difficult to directly translate between teams as DevOps efforts scale. In this article, we’ll take a look at these challenges, and sketch an approach to overcoming them.

The evolution of DevOps

In order to understand the challenges – and the solutions – to using DevOps at scale, it pays to understand the position that your DevOps teams find themselves in. In many organizations, the decision to implement DevOps practices is taken by relatively low-level managers, albeit those who are most progressive in their approach.

In general, this means that DevOps approaches appear first within small teams, and typically evolve from basic agile development techniques. These teams make use of a number of tools and approaches that can make DevOps adherence easier, but all are based on a simple insight: that developers and operational staff can (and should) work together, because their work is mutually constitutive.

This evolutionary approach means that companies on the cusp of applying DevOps at scale will generally possess a large number of small teams, each of which have implemented a DevOps approach in a different way, and with varying levels of success.

The challenges of scale

Scaling DevOps presents a number of key challenges for managers, employees, and leaders alike. Some of these are the consequence of the way that DevOps techniques gradually evolve within organizations; others are unfortunately an inherent feature of DevOps.

First, there is the fact that simple DevOps managerial topologies might be effective at integrating the workflows of small teams dedicated to the development of a particular software product, but they are far less so when applied to a firms’ total software output. Techniques such as normalized production processes make it easier to catch bugs for large firms, but an over-eager application of these techniques runs the risk of homogenizing otherwise highly adapted teams.

Secondly, there is the challenge of inconsistency. Because, as we’ve explained above, DevOps techniques generally arise within small, forward-looking teams, they are generally also highly adapted to the particular needs of these same teams. This can mean that managers trying to standardize DevOps across an organization can be faced with dozens 9if not hundreds) of different approaches, many of which have proven to be very effective.

Finally, there is a more basic problem – that of communication. Where DevOps works best, it facilitates close and continuous communication between operations and development staff. Achieving this level of communication not only involves that communications technologies and platforms are in place; it also demands that teams be small enough that they can genuinely talk to each other.

Most of the emerging DevOps tools are designed to make it easier to communicate, via such interstitial techniques as infrastructure monitoring, continuous integration, and automating the build cycles of other tools and software. However, it’s important to recognize that the challenges associated with scaling DevOps are often as cultural as they are technological.

First steps towards DevOps at scale

Before we get into the specific steps you can take to start scaling your DevOps teams, it’s important to enter into this process with the correct mindset. Or rather, it is important to recognize that DevOps isn’t a separate philosophy or software development process from regular tools, and that it should not be forced upon teams for whom it is unsuited. In other words, there might not be just one way of scaling DevOps, but there are plenty of ways of not doing this.

Generally, teams who have been early adopters of DevOps will see great success, and as a result will get inundated with requests. Skilled managers, aware of this fact, might take the "natural" next step of putting this team in charge of standardizing DevOps practices across an organization. This would be an enormous mistake: the kind of development workflow that works for one product, or one team, is unlikely to meet with the same success if applied in a radically different context.

Instead, leaders should take the time to consult with multiple teams across their organization, and craft a DevOps model that can be usefully applied across all of them. Key to this model will be flexibility: teams should be able to easily see how the model applies to their existing structures, and to see how it will benefit them.

This said, there is no harm – fundamentally – in using a successful team as an example for the other teams in your organization, or giving employees with experience of DevOps a lead role in these transformations. Just make sure that you think of these team members as "thought leaders", rather than elevating them to a managerial position they neither want nor are trained for.

Tips for optimizing success

Beyond the large-scale approaches we’ve mentioned above, there are also a number of crucial elements for scaling any DevOps approach. While the way in which these will be applied will vary depending on your organization, any successful DevOps transition will make use of five "pillars". Here they are:

1. Automate

Let’s get the most practical step out of the way right from the get go. Automation is a crucial step in scaling any process in any firm, and that goes double for DevOps.

The primary advantage of automation for both development and operational teams is the obvious one: that by deploying automated tools, both teams can achieve their goals more quickly and more efficiently. In the context of DevOps, this efficiency is more crucial than ever before, because it frees up team members to communicate with each other, and to improve the efficacy of development processes.

Automation also has a number of less obvious advantages, though. Automated testing and data analysis can provide a standardized set of data for development teams and operational teams to work from, which is why surveys show that 89% of professionals think automating their data analytical processes can help improve the success of their companies.

2. Establish (Small) Platform Teams

For some managers, the way in which DevOps generally develops within a company – on a microscale, and driven by individual managers and teams – will be seen as a problem. Surely the most effective companies are those in which teams are vertically integrated, and in which business insights are applied across whole corporate structures?

Well, yes and no.

In reality, effective attempts to scale DevOps generally involve the establishment of dedicated platform teams, at least for the development of new software products. Research indicates that small teams are usually more effective at sharing knowledge, and are better able to adapt to rapidly changing market and development demands.

3. Redefine Your Goals

Now, we’ll get into some of the more fundamental changes that organizations looking to scale their DevOps processes will have to go through.

Without wishing to be dramatic, adopting DevOps within an organization will involve everyone – from the C-suite to trainee developers – re-defining the purpose of their work. Ten years ago, it was possible for software companies to see themselves essentially as manufacturers – building machines that were then released, and used by their customers.

That approach will need to end if DevOps is going to become successful. Instead, companies (and individual team members) need to see themselves as working in collaboration with their customers. Instead of a customer buying a product, in other words, they are investing in an ongoing relationship.

At the level of internal organization, this means affording operational staff the level of authority, leadership, and respect they warrant. They are not, as many managers still seem to believe, merely the "maintainers" of software that is built by development teams. Instead, they should be seen as active co-creators of software products. In other words, we will all need to re-define our goals, because we no longer build software; instead, we maintain services for our customers.

4. Skills and Culture

Any successful organizational shift is contingent on providing high-quality and relevant training to employees and staff members. However, since DevOps is more of a philosophy than a set of prescriptive techniques, education is even more important when scaling it.

The overall goal of this training is clear enough – it should aim to break down cultural, linguistic, and knowledge barriers between operational staff and development staff. Thus, it should begin with a process of knowledge sharing between these teams.

One of the major barriers for scaling DevOps is still that both sets of staff have some fairly inaccurate ideas about what the role of the other is. Explaining how development works on a quotidian basis, or how much work and expertise goes into operational management, can go a long way to building a dialogue on its own.

Secondly, managers should focus on building a culture within their teams that lends itself to effective deployment of DevOps architectures. Ultimately, organizations should aim to arrive in a position in which employees no longer describe themselves as "developers" or "IT operations" staff, but instead see the two workflows as inherently connected.

All this said, a "cultural" approach of this type can turn off more experienced staff, who may be protective of their corner of the DevOps lifecycle. You should therefore take care to stress that DevOps is not (just) about facilitating communication between teams, but that it can also have dramatic effects on the efficiency of them. In other words, stress to your developers that DevOps will make their life easier, and you will get easier buy-in to the process!

5. Iterative Feedback (Even for DevOps Itself)

By now, most organizations have grasped that a fundamental part of the DevOps paradigm is to seek feedback on (almost) every aspect of a software release, and then continuously iterate this feedback through development and operational teams. There are even a number of standardized approaches to implementing such a system.

However, scaling this iterative process so that it can take place across an entire organization brings its own challenges. One, at a technical level, is that feedback should be shared between all the teams it is relevant to, but it’s likely (especially if you’ve taken the approach we recommend above) that these different teams will be working with different platforms.

In order to ensure that feedback is made as available as possible, therefore, you should use an open-source database with important features such as supporting numerous different programming languages: doing so will help you better manage data integration and scaling processes.

Secondly, it is possible to focus too much on external feedback. Or rather, if you are scaling DevOps processes so that they are being implemented in dozens of different teams at once, it’s important to measure how smoothly this process is occurring in different teams, and to cascade this learning across your entire organization.

Think of this as the DevOps process applied to DevOps itself.

The central insight that DevOps is built on – that software development should be a process, not an event – can equally well be applied to the process of scaling DevOps itself. Moving a team from a "traditional" development mindset to one that incorporates continuous development is not, in itself, an event. Rather, as a manager you should think of rolling out DevOps processes as a "software update" for your teams, and one that will require continual tweaking, rather than a hard reboot.

The journey is the destination

This last point brings us full circle. As we’ve said above, at the moment most firms will have some teams that work within a DevOps paradigm, and some that do not. The difference between these teams – beyond their performance – is likely to be the vision of their leaders. While such vision should be rewarded, it’s important to recognize that DevOps processes cannot just be replicated across teams.

Instead, scaling DevOps should rely on the same philosophical insight as DevOps itself: that everything is a process. Eventually, it may even be that we see the division between operational and development staff disappear. For now, however, two things are certainly clear: DevOps is going to become more and more important in the future, and scaling it is crucial for success.

About the Author

  Sam Bocetta is a former security analyst, having spent the bulk of his as a network engineer for the Navy. He is now semi-retired, and educates the public about security and privacy technology. Much of Sam’s work involved penetration testing ballistic systems. He analyzed our networks looking for entry points, then created security-vulnerability assessments based on my findings. Further, he helped plan, manage, and execute sophisticated "ethical" hacking exercises to identify vulnerabilities and reduce the risk posture of enterprise systems used by the Navy (both on land and at sea). The bulk of his work focused on identifying and preventing application and network threats, lowering attack vector areas, removing vulnerabilities and general reporting. He was able to identify weak points and create new strategies which bolstered our networks against a range of cyber threats. Sam worked in close partnership with architects and developers to identify mitigating controls for vulnerabilities identified across applications and performed security assessments to emulate the tactics, techniques, and procedures of a variety of threats.