A few months ago Benjamin Wootton, co-founder of Contino, shared some the lessons learned talking to dozens of companies about their DevOps initiatives, from their business context to platforms, tooling and skills.
InfoQ talked with Wootton to dig deeper and get an update on his view of the DevOps landscape in 2015.
InfoQ: You mentioned the Dev and Ops divide is a real problem in the organisations that you speak with. Was that mostly in enterprises and public sector or also in startups?
Benjamin: Startups generally suffer less from Dev and Ops divisions because they are invariably running leaner. Everyone will wear many hats across the software development lifecycle and not just stick to one area of specialism. Most people will be co-located and people will tend to just collaborate instead of following rigid business processes.
Startups should also be setup to iterate quickly and rapidly incorporate customer feedback into the product through both automation and the ways of working. This inherent need for speed drives automation from dev to ops, and prevents silos and process from building up.
What I have seen is some startups who are starting to hit scale begin to build a DevOps problem for the future. They feel that they need to grow up and start to put structures, silos, and reporting lines in place. This is funny because I’m often helping bigger companies undo the same in an attempt to behave like startups and go faster!
Outside of startups, the DevOps problem of silos and misaligned incentives seems to be pretty universal. I’ve seen it in businesses of all sizes and across all industries, both public and private sector. IT is just setup in a strongly siloed way around specialisms, and the whole industry needs to modernise and evolve away from this.
InfoQ: Why do you think such organisations are still not investing in solving that problem? Lack of DevOps awareness? Lack of leadership?
Benjamin: I am seeing significant adoption of DevOps in the mainstream this year. I’ve been speaking to people about DevOps for the last 3 years, and it feels like we are at the point that initiatives are moving off the drawing board into real life.
I think this is because there is a burning platform emerging in many industries - they have to do something to go faster and be more agile. As software eats the world, businesses that used to be retailers, newspapers or financial service companies for instance are effectively turning into software companies via the back door.
Unfortunately, they are not setup to behave like software companies - they have technology legacy and are lacking some of the key skills and capabilities to deliver high quality software quickly. This situation is very much driving demand for DevOps transformation.
Regarding organisations that aren’t investing in DevOps or similar initiatives to speed up their delivery cycles, I think they just haven’t realised how exposed they are. Perhaps they don’t realise how the world is fundamentally shifting to a customer centric digital world which is enabled by software. My prediction is that these organisations will increasing feel the pain of their slow delivery processes until they are woken up by the market or replaced by leaders that are more digitally focussed.
InfoQ: DevOps aims at improving collaboration between Dev and Ops. However in highly siloed environments taking the first step can be daunting. Devs might not want to take on added ops responsibilities. Ops people might be afraid of automation taking away their jobs. What would you suggest as a first step to someone (manager or engineer) in that kind of siloed situation?
Benjamin: I like to start by looking for low hanging fruit - things that we can change to get more DevOps like collaboration in place without making big changes to working practices.
For instance, one of the things I like to do is rotations across all areas of the SDLC to get started. Move developers, testers and operations around the different departments for a day of pairing. It might feel a little random having a tester sat with an ops engineer, but doing this, they will start to build empathy with the other functional silos, they will be better informed to do their own job, and can go back and report to their teams what is coming down the line to them.
I also like to replace slow, unwieldy business processes with simple collaboration. Get people to sit down next to each other to hand over a piece of work instead of filling in a ticket or sending a document to each other. The quality of the communication goes up, the cycle times are reduced, and the environment becomes much nicer to work within. A win all around.
I don’t often see people afraid of losing their jobs to automation. People generally get that automation and process improvements frees them up to work on value producing activity, so welcome it with open arms when they have the space to put it in place.
InfoQ: Can it ever make sense to add a DevOps team to build bridges between very isolated teams?
Benjamin: Most DevOps advocates are against the concept of a DevOps team, claiming that it introduces a silo where we are trying to solve a problem of silos. I understand the theory and I used to say the same, but experience has led to me to change this view.
My view today is that DevOps from an automation perspective is now big enough a field to be a specialist area. There are so many tools and the space is moving quickly that people can concentrate on it full time and deliver competitive advantage to their businesses. Having a team of people working with these tools on these type of activities can really work and help all of the developers and testers to go faster, leveraging up their value to the business.
I like to see senior engineers in this DevOps team who bring a DevOps mindset and career experience across dev, test and operations. They can then go out and coach other staff members onto the central automation platform, ideally giving those teams an increasing amount of ownership of the automation.
Over time, the size and role of the DevOps team can decrease, but I still think there is space for a centralised DevOps function. It’s such an important source of competitive advantage that leaving it to a background activity is a missed opportunity.
InfoQ: “Agile hasn't paid off because there is no visible impact for the customer, DevOps is the missing piece.” seems a simple view for a hard problem (releasing frequently to customers). For instance, many enterprise customers' mindset needs adapting as well. How do you increase release frequency if the receiving end doesn't put in the required work as well?
Benjamin: Almost every business I speak with has done some form of agile transformation, delivering new builds of the code in 2 weekly release cycles. However, very few of them have sped up their release cycles to production. This means that the customer generally isn’t feeling the benefit of the new ways of working within IT.
I view DevOps as supporting the last mile - optimising the processes for getting code built, tested, deployed and out the door in line with those 2 weekly development cycles.
I agree it is hard. It’s not just automation, it’s people and process change and this needs supporting from the top if they are going to support this way of working across the entire software development lifecycle. Luckily, DevOps has given us a label to attach this discussion to and drive both automation and a broader IT transformation which should eventually deliver real benefit to the customer.
InfoQ: Monitoring seems to be a big problem still, despite the increase in tooling and talks about the subject. Is that because it's overused, trying to be all things to all people? Or is it lack of employee motivation or skills to work on improving monitoring? Or lack of management support (allocation of resources)?
Benjamin: I think monitoring requires a heavier investment than people realise to get to a high level of maturity. You can put in a tool, but it then needs significant investment to get everything covered right up the stack from tin to actionable business metrics.
The monitoring and alerting solution has to be good enough to gain the trust of the engineers. Patchy coverage or high levels of signal to noise means that people lose confidence, so ongoing investment isn’t made in the monitoring platform and it tends to fall away as a useful tool before it gets to critical mass.
Monitoring has to be baked into the DNA of the delivery process too. Developers need to be thinking about it and making code which is easier to monitor. Testers need to be testing monitoring and operations need to be relying on it and baking it into their day to day processes. Very few organisations are this mature in their use of monitoring, and it is a big missed opportunity for speeding up release cycles through better coverage and visibility.
To fix this, I think we need everything you mention - more resources, more skills around monitoring and a cultural change which values proactive monitoring and alerting.
InfoQ: From your report it seems that many organisations have yet to adopt a cloud mindset (as in dynamic management of environments and higher developer ownership of those environments). Why is that and how can they move towards a more flexible infrastructure management approach?
Benjamin: Cloud adoption is increasing and in my opinion it is inevitable that most workloads will eventually migrate to the cloud. Building and running data centers is simply not strategic - cloud providers can do it better and more cost effectively than any organisation that I have come across. Your resources are simply better allocated at the application layer where there is a competitive advantage to be had.
With regards to more sophisticated use of infrastructure automation, I think a lot of IT organisations still don’t know the art of the possible. They haven’t seen good use of infrastructure automation, configuration management, test driven infrastructure etc, and just need to be educated on what it is and how to be successful with it.
Other teams have not yet secured the time or business case to migrate their applications onto cloud or flexible automated infrastructure. These teams are focused on delivering new application features so do not have enough space in the backlog to work on re-platforming. This might make sense to the business right now, but as speed and agility become increasingly important, many of these teams will need to bite the bullet and move their applications at some point in the near future in order to keep up with their competitors.
One thing that really doesn’t help with any of this is that there is a skills crisis around DevOps. Even if leaders know what they want to achieve, there are not enough engineers who know how to do this kind of automation or have experience at doing in who can make it real.
InfoQ: When would you recommend the adoption of proper tooling for automating the release lifecycle, namely infrastructure config management? How to avoid that the tools exacerbate the silos (by adding a required level of technical skills to handle those tools)?
Benjamin: Different organisations need to start their DevOps initiatives in different places. Sometimes it’s automation, sometimes it’s more people and process and cultural changes that they need to put into place to get started. There is no set template.
When we do get an initiative started with automation, we sometimes do it because it jumps out as a low hanging fruit which we can tackle without major organisational or cultural change. We can put this automation in place and start to get some momentum behind an initiative. We are however always careful not to simply automate a bad process. If we can see the process is wrong we would tackle that first.
I think that choosing tools that support collaboration is a key requirement. Some tools have a lower barrier to entry than others, and I think this is a huge benefit from a DevOps perspective. I would always chose the tool which came out second or third on a feature evaluation if it was easier for developers, testers, operations people and others to pick up and collaborate on.
InfoQ: Containerization is all the hype now but you found very few organizations utilizing or even looking into it at the moment, right? What are the major cost/benefit factors for such organisations looking into containerization?
Benjamin: We really like containers as a tool for deployments - for making them faster, simpler and more consistent within development, test and production environments. We have had projects where we were able to rip out thousands of lines of deployment scripts and replace them with simple pulls and pushes of Docker containers, improving quality in the process. Simplified deployments are reason enough to use containers to us.
Other benefits of containers include more consistency of environments, more density of compute on a single server, better isolation of processes and workfload flexibility across servers and data centres. Containers are really compelling as an approach - anything else feels really retro in comparison!
On the downside, I do think containers are currently more complex to work with. They’re a new paradigm and the tools are evolving quickly, so very few people are experts across the stack. You definitely need DevOps type skills to manage and orchestrate applications that are distributed across many containers, and as I mentioned earlier, these skills are generally difficult for organisations to find.
InfoQ: What's your take on the re-branding of sysadmin jobs into DevOps jobs? And how would you recommend engineers (dev and ops alike) to upgrade their skills for working in or towards a DevOps environment?
Benjamin: We speak with and hire a lot of DevOps engineers, and it is disappointing when people come through who are essentially re-branded system administrators. It’s usually pretty obvious that they don’t have the mindset and approach necessary to break down silos and shift culture, even if they have the automation skills.
That said, I am 100% behind people modernising their skill sets. Just commit to learning what it is all about at a cultural and business change level, and dedicate the time to gain experience with the modern automation tools associated with DevOps. On a personal level, you will be rewarded with career success if you do this.
The tools in and around DevOps are so accessible - most are free and open source for instance. Likewise, the work of DevOps leaders such as Patrick Debois, Gene Kim and Jez Humble is all there on the internet for free consumption in order to learn about the drivers and philosophies behind DevOps.
InfoQ: Do you want to share with our readers any other curious stories or anecdotes from the talks with your customers?
Benjamin: I was surprised to find that it’s often software developers who seem like the least willing to engage with DevOps and want to stay in their coding silo. Across all of our clients, our experience has been that the testers and ops guys are generally keen to engage and improve the path to production, whereas developers are generally focussed on coding and less excited about deploying and running their systems. We now combat this with more coaching and earlier involvement, but it was a surprise nonetheless.
Secondly, I’m really starting to see the working environment as a key factor in DevOps. Our clients with nice working environments full of breakout areas and whiteboards and good facilities seem to have a much easier time moving towards a collaborative DevOps model. These issues sound a bit light for an engineer to be thinking about, but I usually recommend to improve these facilities as a low hanging fruit for getting more collaboration in place.
Finally, I feel like some of our customers are almost looking for someone external to come in and give them permission to start applying common sense and pragmatism to their working practices. They have built up process and certain ways of working which they know are heavyweight, but someone saying that stripping this away is DevOps gives them permission to take a fresh approach. It’s great to go in there and rip out bureaucracy, freeing up teams to work on value producing activity.
About the Interviewee
Benjamin Wootton is the co-founder and Principal Consultant at Contino, a UK based consultancy who help organisations adopt DevOps and Continuous Delivery tools, practices and approaches.
