Engineering Culture Trends Report 2021

Transcript

Shane Hastie: Hello folks. Before we get into today's podcast, I wanted to share with you the details of our upcoming QCon Plus virtual event. Taking place this May 17 to 28. Qcon Plus focuses on emerging software trends and practices from the world's most innovative software professionals.

All 16 tracks are curated by domain experts to help you focus on the topics that matter right now in software development. Tracks include, leading full cycle Engineering teams, modern data pipeline, and continuous delivery, workflows and platforms. You'll learn new ideas insights from over 80 software practitioners at innovator and early adopter companies, spaced over two weeks at a few hours per day.

Experience technical talks, real time interactive sessions, asynchronous learning, and optional workshops, to help you validate your software roadmap. If you're a senior software engineer, architect, or team lead and want to take your technical learning and personal development to a whole new level this year, join us at Qcon Plus this may 17 to 28. Visit qcon.plus for more information.

Introductions [01:33]

Shane Hastie: Good day folks. This is Shane Hastie. I'm the lead editor for Culture & Methods at infoq.com. And this is the Culture & Methods trend report for 2021. We thought we'd explore a slightly different way of doing it this year. I've invited my colleagues, the editorial team from the Culture & Methods area at InfoQ to discuss what we've seen over the last year. I'm privileged to be joined by Craig Smith, Douglas Talbot, Ben Linders, and Raf Gemmail. Unfortunately, our colleague Shaaron Alvares was not able to join us, but she sends her best regards to all of us and to all of our audience.

Team thank you all for joining us today. Let's just do a quick round-robin. I'm sure many of our listeners have read your content but haven't probably heard your voices all that often. So, Craig welcome. Tell us who's Craig.

Craig Smith: Shane, yes, I'm Craig Smith from Australia. I can't remember how long I've been around InfoQ, but it's long time, and to bring the perspective of agility and Culture & Methods from this side of the world.

Shane Hastie: Welcome. And Douglas.

Doug Talbot: Hi, Douglas Talbot. I'm currently in New Zealand. I've just moved back from eight years in Europe where I got to see many of the differences between running teams in different Cultures across the European continent. And it's really interesting being back in New Zealand, comparing the two. And my passion has very much been organizational dynamics and how team dynamics and bits and pieces of organizations interact with each other. And I've recently put out a couple of articles on product thinking, but I guess my other big passion is leadership.

Shane Hastie: And Ben.

Ben Linders: Ben Linders, based in the Netherlands. Very happy to be writing for InfoQ. Started in 2012 and still being impressed and surprised by all the stuff that's happening in the Culture & Methods. Mostly focusing these days on the leadership stuff, Culture change and trying to get case studies, practical stories from people doing Agile in the role to getting those stories out on InfoQ.

Shane Hastie: And Raf.

Raf Gemmail: Hi. I'm another Kiwi. I'm also New Zealand. I've been writing for InfoQ since about 2012. A good friend of mine, Katherine Kirk, who was doing the Agile stream with QCon at the time, brought me in, we did a talk together and it led one thing to another. And the funny story is I was put in touch with Shane and Shane was my lead editor about like, "All right, whereabouts in the world are you?" And we just moved to New Zealand and it turned out Shane lives 10 minutes away from me.

My passions are here, Engineering, applying Agile Engineering techniques, XP methodologies, which I still think have a lot of value. I've been coaching Agile teams, coaching technical teams within the technical practices. And it's led me through a journey to teaching. Currently, I'm a head of product with a company in the education space. I think that's going to connect with the conversation today because I see a big sharp rise in how we value education, how we do education, how those educational skills apply to us as coaches as well. So big on that.

And our team topologies is the other passion for me at the moment. It's something which I'm throwing out and just about every other conversation, because I see it as an answer to why Conway's law works, a good way of justifying that when you're having those conversation.

Shane Hastie: Thank you. Welcome. And I can say that Shaaron is on the West coast USA, so we're not all based in the Southern hemisphere. In the virtual world that we live in today, physical location doesn't make a big difference. And that I think is a nice segway into talking about what we've seen over the last year. Obviously, the impact of COVID-19 has been huge in the Culture & Methods space and huge on the world in general. But gents, what does that really mean to us, looking at the trends in Culture & Methods for InfoQ for our audience?

The impact of COVID-19 – forced remote work caused a lot of disruption and bad experiences [05:19]

Ben Linders: Let me jump in: what I've seen is that a lot of teams have been forced to go remotely, to start working remotely since they're not able to come over to the office and they're basically mimicking stuff that they've been doing on site remotely. And I think what you can see, a huge difference in there are these people already capable of collaborating in a different way when they are working remotely in there.

I see organizations who are having meeting after meeting, working remotely, really wearing people out there and people being exhausted already halfway today, and then having to work than how they used to do it. But I see other organizations that are really managed to set up a way that people can work remotely, but also can work autonomously and work at their own pace and bringing in their own knowledge and their own experience in a way, and at the time that suits them well. I think this is making a big difference in those companies where people are able to do that.

Craig Smith: I think coupled with that is the digital transformation that organizations have had to endure. So organizations that might've been holding off on that were forced to go down that path. So not only have the workers themselves been forced to work differently, obviously as customers, there's the force work differently. And then as the services we give to our customers. So it's forced a lot of the methods in relation to thinking about service design and product design and putting those things and applying them in a different way has meant that a lot of things that, perhaps some organizations had a lead on, everybody had to fall into the same field.

Ben Linders: Raf, you were going to say something.

Raf Gemmail: Along similar lines. I saw the same at the very start of the pandemic, which was the attempt to apply the physical workspace metaphor to the remote workspace. And it doesn't work. It was attempts to have too many meetings, as you said, it was misuse of white boarding tools. It was trying to figure out how to facilitate things, how to collaborate as a team, the big slack with too many people, going back to the team topologies thing. There's a good piece of advice, which is don't do that! Have clear groups where people actually do need to interact and communicate.

Coming to terms with remote tooling [07:21]

Raf Gemmail: As you point out Craig, I then saw this need for people to figure out how to use their tools. Well, rapid experimentation, figuring out which are the right tools. Do we use the Miro’s? Do we use the Jamboard? Do we use something else? How do we do remote trust and gathering sessions? How do we do that remote drink and make it effective? Companies which were on premise starting to move more readily to the cloud and trying to make that happen.

There's definitely been this acceleration in what we're doing, how we do it and the remote work thing’s also been an interesting balancing act I've seen, because I was remote working the prior year and there was a lot of the synchronous, but then in post-COVID, the company I was with at that stage went straight into remote first. And there were questions about what that meant. There was still the notion of the nine to five day coming to terms with actually, it's probably better to give people space in their day.

Examples of good remote education [08:11]

Raf Gemmail: There has been the journey through the air. I'm remote teaching and institution I was with, I probably wouldn't have joined it a year ago. But now I'm building a product which is a remote first product. They had an on-site office in Whangarei. And their students are doing really well. It's bringing in people who don't have the freedom to study during the working day. People are able to live and balance their lives slightly better. So there's this emergent effect coming out of it, which perhaps has a better quality of life, which feels strange saying that in this situation where COVID is taking so many lives.

The challenges of never leaving work [08:40]

Ben Linders: I would like to say though that the way we've sort of painted it, there has been quite rosy and positive that it's all been steps forward. But I think that's definitely not true of many organizations. Yes. We can point out exemplars who are handling remote very well. But we're also seeing people getting into this scenario where life is mixed with work and you can't differentiate where the two boundaries are and work carries on the whole day and people never escape it because all of their gadgets are their work life and their personal life now.

We're also seeing large groups that might've come together and done anything from iteration planning or workshops together, really struggling to come together and interact in a remote world because everyone's a little dot on a screen and you have to wait turns and such like. So the interplay and the random conversations in the washing machine effect amongst people is now being lost. We're not seeing that same brainstorm and collaboration effect. And yes, we're trying to simulate and find ways around those. But in my opinion, they haven't reached the same height as we were getting out of that face-to-face work.

And as new people come in, they really struggle to socially integrate into the workplace. They're really struggling to get a water cooler conversation or a coffee or go to the pub. And so all of those elements of getting lost. So I do think that there's a lot of growth to go on in this place and finding a happy medium between a remote only style organization, which I'm not really a believer in at the moment and some balance or the forced aspect of COVID is going to take us a lot longer I think.

Raf Gemmail: I wonder when you say that, because I kind of had similar doubts as I was seeing this and the company I was with at the time, going through the journey. And there's a thing in my head, which is that their outputs, their behaviors, we want to see in the onsite teams that we see, we trust collaboration first. I don't think this would have happened 10 years ago when the tech was not as good. But if we focused on those outputs, how do we build that trust? How do we get the output from that collaboration session?

What it takes to create a good remote environment [10:32]

Raf Gemmail: We're able to facilitate those outcomes in the remote setting. And I think it does take effort. We're going through the learning period now, but I think it is achievable. And I think at the end of that journey, I mean, there's a personal bias. I love remote, but I think at the end of that journey, there is an alternate paradigm for how we work. And there's a better work-life balance and people will be going through there.

And maybe that's part of why we're here. We've got this curve, we've got the innovators and we've got the early adopters and there's some people are going to have an advantage because they'd started there and they have a bit of a bias and they've seen the Gitlabs and how they're working out there and they may have done it themselves.

Hybrid working - the worst of both worlds [11:05]

Raf Gemmail: And then you've got people who are early majority or late majority and the term being thrown around a lot right now is hybrid working. This mix between in the office and remote. And I can see people are still holding onto that. I hear people wanting to hold onto the workplace.

And my experience with the hybrid model when it was starting was it was actually really bad because there was this disparity between people in the office and people remote. It broke the communication. And I think it was one of the news pieces I did on a company in Cornwall. And they were looking at how they could use their onsite time better towards trust building and such. So, I hear what you're saying. I'm hopeful and optimistic towards a good outcome, but I think we can lead the way. We can coach the right areas.

Shane Hastie: Craig, you have something to say on this.

Craig Smith: I think it's also bringing out some of the worst though. So, as we start to hit, well, I guess we're calling the new normal or whatever it might be, but now we're sort of been dealing with this for every year. The amount of organizations now that are coming out with these forced hybrid models. So you must be one day or two day in the office, which also then for every organization like Spotify or Facebook that might come out and say, you can work remotely forever.

It's putting an interesting spin now on how organizations actually treat their employees and the Culture that is behind them. And I think what's been interesting is we think about, from models where we're Agile coaches on this journey, the difference between trying to do things like support organizations on their journey, when the only thing you can see is through a zoom window.

The challenges of forming new teams in a fully remote environment [12:25]

Craig Smith: I was speaking to people early on in the pandemic and they go, "Yeah, this has worked fine." We just pick everything up and we just put it behind a bunch of electronic stuff and we put everything on electronic whiteboard and where we went. And I think that was fine because we all had this connection that we knew how everybody worked in the office. And it was just, instead of you sitting three chairs from me, I was just seeing you in a zoom window. And what I've found interesting as then, as I've started to work with organizations or teams that would have formed post-this, that a lot of those cultural norms, the effort that we might've often put in to get people together, to learn each other's differences and how we might work together has changed. And it's shown divide a little bit between organizations that have embraced this way of working versus those that perhaps are trying to pull back to that old style of work.

Shane Hastie: So what I'm definitely hearing here is there is good remote and there's bad remote. And we've seen examples. We've seen situations of both. So if we think about our technology adoption curve, I'm going to say that probably over on the left in the innovators early adopters are people who are doing good remote. My guess is the majority are really struggling with that. What advice would we have to what makes that good remote?

Advice around what makes good remote [13:34]

Ben Linders: I think it's a famous story of not having one size fits all in there which is really important. And I work with people all around the world on the work that I do for my own company and also on the horizon that it will reproduce.

And a very different ways that I work with people based on what's their experience, based on the way that I would like to do that work. I think what's given me a benefit in that it's the flexibility to adapt to somebody else's style and to find a way that's working for us in there. And I think the other that's given me a big advantage in there, is I basically start from assuming that the person I'm connecting with, that I'm starting to work with, wants to do a good job and want to go to watch a good result the best way they can. And then we tried to find a solution to get there.

Starting from that relationship can get you a long way. That's a key issue, which the nominate works good in a shortage situation from what's not working well.

Raf Gemmail: You're connecting with the conversation I had last night. So Katherine who I mentioned earlier, Katherine Kirk, she does a lot of talks in the politics space, but she's got this little group and there's a book coming out at some stage. But she has these models that a bunch of us friends share and apply in the workplace, which kind of lenses to see what's going on in an interaction. I used them a lot in the remote space. It's like trying to see where the other person is coming from. What is the context? You see the pain, the struggling, someone who's having trouble there. And yeah, things like check-in protocol, maybe examples of this, understand where they're coming from.

Lots of lenses, like what are the tensions in this context? Is this person working from home in their kitchen with other stuff going on? Is this person someone who's been in back to back meetings for the last four hours and these that zoom fatigue and it's then being able to call, should we just do this later?

That empathy plays a bit because we're not in the office sharing each other's day, trying to understand the context. This person's on three projects. And can I go and co discover with them what it is that they haven't done. Can I look at their current backlog and just feel if I'm adding more pressure? I don't know, there's this thing I'm slightly more aware of, which is building some contexts, which maybe me spending more effort. We have these lenses. The reality is this notion that order falls into disorder, just because someone is there last week and everything seems to be running smoothly, but be aware that that human context is changing.

It's all to do with entropy and change. You've got to just be aware of that, because I don't have the continuous real-time cues. It is about building a compassion when you're in those conversations. And in those moment.

Shane Hastie: Quite a lot of research and work in looking at the psychology of this and trying to come up early on when we were covering a lot of the COVID stuff. So what are some of the things that you found?

The psychological impacts and resources to help [16:09]

Doug Talbot: I did a few articles with Dr. Michelle O'sullivan. She's a clinical psychologist that I got to know in the UK. And we were talking about mental health in those articles. And I think that we're still seeing mental health as a key aspect of this whole process now because that tangible human interaction has vanished, particularly in Europe and other countries around the world.

New Zealand is incredibly privileged in this space at the moment, because that physical interaction has disappeared. We've got to be far more aware of the human interactions in our organization, and we have to make time for them. And that means if you're doing some form of leadership in your organization, then you now need to think about, how am I going to have my tribe of 150 people interact, socialize, connect, get to know each other so that trust can be there and empathy can be there. And spotting when someone's having a bad day can be there, which you might've picked up body language before. But now that you're only maybe seeing someone in the corner of a remote zoom window and going, "Are they acting differently? Should I ask a question? How do I get that one to one with them to ask that question now as a leader."

All of those things have become much more essential to dedicate specific, structured time to achieve. And we just used to have it almost happened by accident on a walk the floor or at the water cooler.

I think the good remote is where there is focused attention on the human interaction in your organization. The bad remote is often where they haven't made a structured attempt to build that into stuff that's now very process driven.

Shane Hastie: Thank you. And I know we could continue talking about this one for quite a long time, because we're all pretty passionate. But if we think of our trends report and the technology adoption curve, I think we could say at this point that there is the good remote. There is the bad remote, and there are some ideas to help move those good remote practices into more organizations that we hope. So what are the other things? And Craig, you touched on the digital transformation and the acceleration of that. What does that mean to our audience?

Digital transformation is now late majority – forced on almost every organisation [18:11]

Craig Smith: Well, it means as customers, what we might expect from our organization, things that we've been talking about for a long time on InfoQ, things like Conway's law and the way we see an organization, there's a lot more exposed. There's a lot of work. I know Raf, you feel a lot of passion around, team typologies and those type of topics that's become even more important because the way we build the way we collaborate is what now starts to expose itself to the outside world.

And similarly, when we come back onto the inside of the organization, the fact that we've all been forced to be cloud first, we're recording this podcast in a cloud-based format or using cloud-based tools. We mentioned things like being forced to use tools like Miro and Mural. DevOps in some ways of forcing itself on top of us. But what it's now doing is starting to really be a bit of a forcing function to where are some of those roadblocks or alerts inside our organizations of things, where we haven't quite perhaps grown up or gone as fast as we need to, as we try to compete in this new world.

Using data to generate insights – AI can help, but it needs people to apply them [19:08]

Raf Gemmail: As a connection to that, you talked about the alerts and the alerts implies observability monitoring. I've seen that there's more automation, which leads to more understanding of what's happening in our products. And we're getting deeper insights. And those insights I'm seeing, are leading to questioning how we can get a deeper understanding of that data. So there's a connection from there to things like using AI models, to get a deeper understanding of how your products are being used, how different metrics cross correlate, being able to bring those back into product decisions.

I mean, we're already moving there. We're doing multi-variate testing and all sorts of things. I've seen the whole flow, the whole value stream result in more data and having more data across the value stream means you can look at those early engagements from HubSpot in marketing, and you can cross correlate them with sales output and with quality and meantime to failure in the middle. And all of those metrics are there to be exploited a bit more.

Ben Linders: I think that the tools are there and certainly AI looks very promising and as a way to dig into this data to really get these insights in there. But I think the only starting companies who are really courage enough to find out how the products are being used and not really being interested in having to Culture in there, to learn about this stuff. And they're asking the right questions and are willing to ask the right question, kept this understanding in there.

I still see myself further with a lot of tickets for the fair. I'm really thinking like, "Okay, I made this mistake five or six times already." This should be to dig out that there's a mistake in here designed to take it there. And that probably not only customer bumping that there.

But if they're not looking for that insight, they will not ask you that question. Not looking to take out that information from the data. They're not getting it. So the tools are there, but it also needs to Culture and the drive and the people to find out to really get the benefits out of it.

Raf Gemmail: That's very true. You can have the data, but you have to ask the right questions of the data. Maybe our questions are still biased by traditional models. There's a product doing well. I'm still talking about failures. I'm sure that deeper insights, there's a tool Plural sight flow. It's this thing that gives development teams, deeper insights into how their developers are developing.

Things like automated heat maps of where technical debt is. If you're working on an issue, how much of your issue is debt? And that's digging in for insights that you can then use to prioritize where you put your technical debt and tools like that are becoming more prominent. I don't know if they're connected with remote and the COVID, but asking the question is hard.

AI systems as team mates [21:34]

Shane Hastie: So last year in the innovators space, we did identify AI and ML tooling, what I'm hearing and I'm certainly seeing some articles on socio-technical systems and on AI systems as teammates. What I'm seeing is that that is coming forward. Is it enough that you would say it's moved from the innovator to earlier adopter or are these things that are only being used in the innovator space at the moment.

Ben Linders: My impression is they’re still innovator, still rarely used in there. And all these kinds of users are more or less individual users. Individual case, one company using it in one certain way. And there's no shared learning yet in this area. I'm hoping that we can drive this on InfoQ.

I tried ... There a couple of articles there, on companies that are using AI in the processes, and I'm hoping that other companies are picking up on that and building on top of those ideas to experiment and to do their own ideas. But I don't see that happening yet, it’s used to solve individual isolated cases right now.

Doug Talbot: I think we need to be careful to distinguish between true ML and AI models and just technologically enabled measurement. We're seeing a lot of the digital transformation bit that Craig brought up where we're being forced to now use tools to measure remotely distributed actions across our organization. And suddenly we've seen this even more burgeoning of these Culture, I guess, engagement checking tools, where they poll the organization every day or the question and stuff.

But no one's really intelligently using that data and really understanding what it means and because it's highly complex data with huge numbers of moving variables and definitely way out on the complex space and they can even kind of grid. Then I think at the moment, we're just almost spamming ourselves with data and not really knowing what to do with it or using it really badly in our organizations.

And then we crossed from that into, well, how can we really use the ML kind of tricks that they're using for instance, in cybersecurity, where they're getting algorithms to really learn the patterns of attack and spot them and turn them off before any human could have ever intervened. We're definitely not applying that kind of level of learning our people and culture space at the moment. So that's really early innovator for me. And I'm worried that I'm not seeing anyone using the kind of digital transformation pieces in that space really intelligently. So that's definitely in innovator space.

We’re flooded with data, but not generating the insights needed [23:58]

Raf Gemmail: Yes. I agree for Culture and application in this domain, we're still figuring out how to use that data. I've seen the application of models that provide insights into people, how they behave and such and conversations about how do we use this. We know who our customers are, but how do we use this? We know things about them. We have classifications, but we're still at that stage where we're like, "Wow, shiny magic insight." But yes as you said Ben, what is the question? This is answering it. You kind of retrofit the question to the answer.

Doug Talbot: Even the accelerate measures like the meantime to failure and stuff that have become really popular and got really obsessed by our Engineering types. They're a fragment of a value stream. They're not actually telling us whether or not our value stream got better, or we got more productive as an organization. They're very much an isolated element, just looking at software. And I think that that's a mistake to start narrowing to that space.

Raf Gemmail: You can't cross correlate. As I was saying that you can look at your marketing data. So I've done that sort of thing. With Splunk that if you've got the data that comes back to the question, what does it look like? If I plot my marketing data with my product data and I can see how they cross correlate, I can correlate it with my release events and the levels of insight there. So I think there is data there, but the question isn't always obvious, but what I would say is that we're in a place where we're collecting more data than before we value data. We know that every event is valuable. We have been throwing away a lot of information for a long time and we're moving to a place where we can stick them in a data Lake. I can stick them there's three buckets and build a data Lake out of it and then analyze it later.

How much of it gets used? How much of it is valuable? I don't know if there's a role for that.

Doug Talbot: I think part of this is for organizations, the smart ones are able to take that data. But what I'm seeing in a lot of organizations is that we still treat data as a separate silo or department. And so now there's this channel because we haven't culturally only built data in as being data first as part of our teams. When you go back and read the XP books, and then they talk about the tester and the developer and the analyst, but there's no mention of the data analyst or data engineer that we might need in order to make sense of all that. So what I am now seeing in organizations is this queue. Now people coming towards their data or their BI, their departments, because they need this type of thing. But a lot of our engineers don't have that experience in there. And it becomes now a throttle for organizations.

So we're collecting all this data, but we now need to get better in the way we culturally start to integrate that into our Engineering processes.

Shane Hastie: So we're talking about, we've got all of this data, we can access all of this data. We can use all of this data. I want to throw up the question. Should we? One of the topics we've been exploring on InfoQ for the last couple of years is ethics. How are we doing in terms of Technology for good and the ethical behavior in our industry? Where are we headed? What's happening?

Ethical implications and challenges [26:41]

Ben Linders: It's there. It's definitely there. And I see more and more companies becoming visible that are actually doing it. Not just like picking it up. It's companies who've been doing this for years and looking at tech for good and looking at, "Okay, why are we running this company? And our goal is not to make money for our stakeholders or we've got a different goal in there." So I think the challenge that we have, I see a few innovators in there is to show this to the world that is actually possible because companies are doing this.

Craig Smith: I agree with you. I wonder though how much that actually filters down to the people on the ground though. So I see there has always been, the big issue of Culture is, we can put our values up on the wall and state that we apply for, trust and integrity and honesty and doing the best for the customer. And that's the focus.

But the problem is that what I am seeing in all of this is that when you get down to the engineers and all the other people who are building out products is they're under increased pressure.

And I worry that the ethics down at that level are being sacrificed because, do I call this or do I just get that ticket out the door? Because it's the next thing in the queue? We're saying yes, more often to all of the work. Can you work on seven projects at once? Yes. You know, can you get this thing through the whole thing about this invading into our home life?

And so I think on some of the big issues as a community we've seen over the last year, the community rising up around things that perhaps have been bubbling under the surface for a while. But I also still worry that the pressure that we're putting on people is that when you actually come down to the people who are actually doing the work down at the team level, I think we've still got a long way to go, to allow people to feel safe. That whole psychological safety that we always talk about in the Culture of teams. I'm not entirely sure is there as much as it should be, even in those organizations, sometimes that are proclaiming that they're doing a good job of it.

Ben Linders: I hear your worry.

Real psychological safety is hard to achieve [28:25]

Doug Talbot: I'm seeing a huge amount of literature, discussion and dialogue about psychological safety and those kinds of things, Craig, and I'm seeing the implementation in the organizations 99% of the time fail to understand it and fail to deliver it. And I think that there's a real issue. And that's why I was mentioning is Culture engagement, polling tools, and such like, as people are starting to pretend that that data is enough to understand whether they're doing the right thing rather than creating relationships between leaders and their staff and really building the right kind of empathy and environment to really thrive.

And I think leadership education is one of the big things that I think is starting to emerge into our innovators space. In fact, I think we even had something about it last time. I haven't got the graph right in front of me. What was the thing about leadership in there, Shane?

Shane Hastie: Systemic and leadership coaching.

Doug Talbot: Yes.

Shane Hastie: We were saying that was an innovator practice last year.

Systemic and leadership coaching helps [29:22]

Doug Talbot: Yes. So I am seeing more organizations start to commit to actually thinking, how do I bring my leaders forward to enable psychological safety, to enable this idea of motivated, engaged people, rather than just relying on a once a year engagement poll or now a once a day engagement poll and pretending that the data is going to somehow fix something. So I think the next step I'm seeing is active, educated, researched education of our leaders. I'm seeing now all of the courses come out and remote has triggered this as well from Scrum Alliance and Agile and all of these groups where we're now seeing leadership courses, people development courses, which are really talking about how do you bring psychological safety to a life properly rather than just pretending that an engagement campaign every year we'll do the job. So I'm hopeful, but I think it's really poor at the moment.

Ben Linders: I agree it needs to be developed further. And what I'm also seeing. And then we had some stories about that, only InfoQ also, leaders who are sharing how they are doing this in their organization, how they all working with the people, how they are coaching the people, how they're being vulnerable themselves, about the mistakes that they've made in there and why they're doing this.

Raf Gemmail: That they're really doing a shout out. This is important and they're sharing their personal stories in there. I think those kinds of personal stories can also trigger other leaders to either share their stories and speak up as they are doing, or at least make them think about, "Hey, the value with this kind of stuff and daring to experiment with this."

Shane Hastie: So what about things like diversity and inclusion? How are we doing there. There's been a lot of, dare I say, "Lip service?" How well are we doing?

Diversity and inclusion is improving and we still have a long way to go [31:01]

Ben Linders: It seems to be moving towards the inclusion part, which I think is a good development. There was a lot of focus initially on diversity, but just having people invited them, creating the party where the people want to be done. It's not going to be working out any way. So I think that in the sense is a good development. But still, I think there's a lot of lip service there also. So how many companies are truly doing this?

Doug Talbot: I think that we're seeing a lot of great intent, the Black Lives Matter movement. And I guess just generally trying to figure out how we can enable interaction of a wider diversity of people now that we have remote access to the entire globe of people to be part of our organization has changed the way people are looking for people even now.

But the issue that I think everyone's struggling with is, it's not just a matter of hiring these people, just suddenly switching and saying, we're going to change our basic hiring model. We've got to also change our internal Culture to allow them in and allow them to be treated as an equal in our organizations rather than having an antigen effect, an immune effect and rejecting the person who's speaking out and saying something different.

So we're going to have to change that, but we're also suddenly finding actually a lot of these people, we don't have an access mechanism to them. I mean, the lack of people with STEM skills in our industry has been something that's been on the table for decades now, and we still haven't solved it. And now we've got the problem of not only do we not have enough, generally educated people for our industry, but also the full diversity of people that could be in that industry are definitely not there, but that's still a decade away to develop that, to bring it through schools, to change the Culture in the country that we're trying to hire in.

It's a much bigger, wider, systemic problem rather than just change your hiring process. And this is not going to be an overnight fix. It's probably going to be on our curve for a long time because it's generational national level or international level, not just organizational level.

Effective remote learning can help bring disengaged and disenfranchised communities into the tech industry [32:56]

Ben Linders: I think actually that remote learning is something that can help us here, because if we have to wait until these kinds of things are brought into the curriculum of universities, it's going to take us a long time. But now we have a virtual course has much more opportunity for remote learning and also possibilities there for people to use that kind of learning, to develop themselves who are not able to go to university. So I think the expansion of learning possibilities can help to drive this and can give a boost to this.

Raf Gemmail: This is something which I'm seeing in the last two years, I've worked with several reinterpretations. People disrupting the university paradigm. People still getting degree qualifications, still getting that high standard learning. The people coming they not the people changing their lives. And that's where I'm seeing the diversity matters. I'm seeing a miner, currently a miner, as in someone who worked in coal mines in Western Australia, coming in to learn IT skills and that changing their lives. They're bringing this wade range of experiences, which I can see in a team just changing the way we address problems. They're learning the industry relevant skills. They have shorter programs, programs that are better fit into their lives. They're able to come out and join that workforce.

I'm a bit more optimistic about the rate of change. At personal experience, as someone who is of a non-European origin, I've seen across my life, that distribution change a lot. I've seen the industry change from a few people who look like me to many people who look like me now. I don't know what that's worth.

The gender balance has been slower to change, but through what I'm seeing in education, I think, and I'm optimistic that that's going to be something that we see changing in the next few years. And the thing to remember is that going back to this fourth industrial revolution stuff, the rapid ramp up of technology. I think we're in that rapid ramp up and we're seeing it in COVID. Things that would have taken a lot longer to change culturally, previously, and I think the optimistic. I want to see the bright, shiny future.

But I think there is this rapid change. So, education we're making it more accessible. There's a lot more of it, but these massively online classrooms making education available globally.

Doug Talbot: That's great to do. I'm really excited to hear you say that it's good that you're in the industry and seeing it. It's fantastic. I'm just worried that it's going to be a long slog and that people went on the boil very quickly and then felt to me like they came off the boil really quickly. And I hope we see an ongoing commitment.

Raf Gemmail: I think the commitment will come out of necessity. So, what I see with remote right now is remote is opening up market places like remoteok.io. And you go then, you look at the roles out there. And that teams globally looking for people globally. And to that's talking about international teams. I was lucky to work in companies that which were progressive, which intentionally sought international teams.

But there were often yeah, people that have the opportunity to have that high class education in their countries. Now we have markets opening up where you can get the person in Poland, you get the person in India, you can get the person in the US, wherever, working together.

I mean, I'm sure if you looked at the demographics of the GITlabs and GIThub right now, you'll see that it's good talent worldwide. And I see that as democratizing. It's making our industry open to many. How we succeed? I don't know.

I think personally that, that async paradigm, that remote working paradigm, might be something we see. And I don't know how that's going to result in cultural fit, how it's going to result in people over process. I don't know how that's going to work out how that balance is going to work. I think that's going to be one of the challenges.

The importance of the underlying culture shift needed for effective remote engagement – beware the organisation antibodies [36:18]

Doug Talbot: I guess I've just seen recently, I've been working with an organization where they have been trying to grow their team quite significantly, and they've been trying to take on the digital transformation. But it wasn't fundamentally part of their culture at all when this whole COVID thing hit. And the result has been that they've brought in lots of excited, new remote people with some diverse skills and then the organization's immune system has kicked in and kicked them out just as quickly.

And so I'm really hopeful that that's only going to be a single example and that there's more winners than losers out there. But it's still really challenging for organizations to actually shift their Culture when it has become so locked in a particular pattern.

Raf Gemmail: I think this ties in with what we were talking about earlier the late majority, the early majority in this hybrid model where they're trying to figure out, the good remote bad remotes. And I think there is going to be, as with anything the trickle down. But I'm kind of optimistic that in the current time, given the current necessity, that that trickle down will be amplified. I've seen government departments thinking more progressively around things like DevOps, about how they do remotes. And having seen some government departments that's quite revolutionary.

Shane Hastie: That's a nice Segway, if we may. Craig, what are the points that you were making before we came on here. Was bringing some of these ideas of agility into government, and we're also seeing business agility. We put into early adopters last year. Where is that at? And what's happening with agile and government?

The slow movement towards agility in government [37:46]

Craig Smith: There has been a slow movement towards agility in government for a number of years. So this isn't anything particularly new, but it's probably been more in pockets. We've seen the US for example, I think just recently put out a 250 page report on essentially agility in government, which is kind of ironic in its own way. That sort of mandating of work practices. This digital first thing led by the UK government and the gov.uk is something that you're seeing spreading. If you look around the world, Canada, Singapore, Australia, and New Zealand all now have variations of that.

But I think what's happened in the world over the last 12 months has started to do is that, because governments particularly have now been, having to prop up certain parts of the economies, and is that this crunch, and we haven't seen this yet, but that money's got to come from somewhere. And sometimes I think the amount of waste that goes in their processes, I lump large corporates into this process as well, but particularly governments as they have been sort of propping these things up, what happens when that support stops? What happens when that funding stops? And what does that look like in the next one, three, five, even 10 years when governments are looking that far forward.

At the moment they have to be on the front foot and it's forced them down this path, but for a lot of them, they've been scrambling to do this. This has been a lot of hard work by a lot of people and you can't sustain that type of process. So whilst I agree, there's lots of good things happening that they're actually talking about DevOps, and they're talking about these collaborative type efforts.

It's only sort of scratching the surface that actually there's going to be a need, because what I worry about is that financial implication in years to come, there's going to be this crunch that the countries that are going to come out of this with strong economies are going to be those that in themselves, start to embrace this ultra internally.

Doug Talbot: Do you think there's a difference between what you're seeing governments take on in the way they're adopting agility and the way private enterprise did 10 years ago? Do you just see it as a delay of the same pattern? Because I guess when Shane asked about business agility, I definitely don't see the business arms of government bringing any agility in yet, it's all been still focused on the tech space and on the engineering space. And I think that large organizations are mirroring that pattern. Any thoughts?

Craig Smith: So interestingly, what I've been seeing more of late is actually the opposite to that, is actually a flip where the organizations now... When I say this, it's not true business agility, I completely agree with you. There's a long way to go in actually that whole cycle time of idea to customer. There's a lot of things to do in that process.

But what I'm seeing now is that almost the IT organization, particularly large organizations are so overloaded and they haven't been able to adapt these things like how do they work on the seven or 10 projects at once that the business is trying to run?

And so I'm actually seeing a lot of push from the business side to go, and we need to do this faster. And so they're picking up the practices, but actually because the organizational culture and the organizational processes behind that don't support it. It's now actually pushing that back into the IT area that are now falling back into old ways of working.

And so they'll now push back and go, "Okay, we can do that, but go and write me a document in order to tell me what you want." Almost to stem the flow of what they're after. So this whole thing of actually working together closely. But what that's now doing is essentially pushing the entire IT department back into those old ways of working where, we'll delay the conversation. We'll delay all those things. We'll do what we know in order to push you backwards.

Breaking the silo model of organisations – moving to value streams and product line thinking [41:01]

Craig Smith: So again, the smart organizations, I see this a lot in large corporate and government, but actually even in smaller organizations that are just trying to make sense of all this. We still treat IT as a silo. We still hear the business versus IT type of a thing until we actually start to think about those team typologies and break those things away, really that true product centric model.

A lot of organizations have been able to put a facade over the front of that to a certain extent as a project or as a breakaway team over here. But actually they're the big challenges is actually, "No, we actually have to do this as a real culture."

Doug Talbot: So I'd like to then propose Shane that we put an item on our innovators early adopters, which is true integration of business and tech and breaking that silo. Not even two items collaborating, but integrating them together fully into service streams, or however you want to describe the concept.

And I think that is something I am starting to see the real innovators start to do well and put together, like Craig said, it's one of the big barriers to overcome, to get true agility.

Raf Gemmail: I was with a government agency at the start of the year before the pandemic. And it was by no means an innovator or early adopter. Possibly early majority, late end of the late majority and many outsource projects. So we're going through a thing where they actually recognized they wanted more product teams. They wanted more product centricity. They wanted to break down the silos and the silos existed. The silos were a cause for a lot of the tension.

And I think there was an awareness, that I was working with heads of the product team heads of the project management team to see how they may untangle that. But even them having that conversation, having that drive, listening to alternate ways of working, recognizing the dysfunction was something I think is indicative of something moving to the left a bit more.

When you're aware of the things that you don't know or the things that you're not doing well, maybe that's a sign that it has actually crept back a bit. So there's an awareness. I don't know, awareness isn't the same as doing, because the doing wasn't happening.

The lack of safety as a huge blocker to real culture shift [42:55]

Ben Linders: There is one big blocker that I still see if I looking at adopting agility in the government world in there. And that big blocker is that there's not really the safety in government to reflect and to learn from things that have gone wrong. And believe me, there's lots of opportunities to learn. If I look at the Dutch IT system, the Dutch government systems there’s lots to learn, but I'm guessing this is worldwide an issue.

There's stuff to learn. And the basic process, I still see if they start to learn this, they wait until the end project two or three years, four or five years. And then when they investigate the project, what you see in there that are looking for who to blame. And you can't get this out of the government Culture. It's always looking on, "Okay, somebody's got to take the blame in there because things have gone wrong."

And I'm not sure how you can get the turnaround in this process, this kind of mindset. Because I think this is truly blocking any true agility, if you look in the government. If they don't get to the culture like we want to learn from failure and we want to see a different way in there. We don't want to put the blame anywhere because that's not going to help us. And it's not going to change that.

I'm pretty pessimistic on this one. And you know am a big optimist. In this area, no, it scares me to death.

Shane Hastie: Ben, if I can pick up from you on the reflection, one of the things that we know you are well-known for is retrospective practices. What is the state of practice? What is the state of art with that reflection, learning and retrospectives today?

The state of practice in retrospectives [44:26]

Ben Linders: I think the state of art is that companies are getting better in collecting the data or getting better in generating insights either by using different formats in the retrospective. I think also partly about using more of the online tools and that there are many tools. I see new retrospectives to open up every two, three weeks, probably not seeing all of them. It's even growing faster than programming language probably.

But if you look at really bringing it back to shared learning and taking action, that's where still the big challenges in retrospectives. We have the data, but getting people to agree on, "Okay, well, what do we learn from this? And what does the thing that we're going to be doing differently right now? What's the thing that we can experiment with taking that to action in the Dean." That's where still, the big struggle is in retrospectives.

And I think one technique that can actually help in there, is a kind of gamification where you can create a setting where people can play to get a better understanding. To get this shared understanding. And once they start seeing the problem together, finding the solutions is going to become a lot easier in there. So I'm hoping that that can make a difference in there.

Shane Hastie: There's one last topic I'd like us to explore together. We are in February 2021. It's almost exactly 20 years since the Agile manifesto was written. That there's a lot of other events going on around, but what is our reflection on 20 years of this Agile thing?

20 years since the Agile Manifesto was written [45:57]

Ben Linders: Well, one reflection that I would like to add is, if I look back 20 years from now, when the first stuff start popping up around Agile, seeing the papers that Alistair Coburn was writing at the time about collaborative game and seeing how things were looking at short cycle delivery and learning and improvement in there.

My first impression that time was like, "Yeah, this is great stuff. And this works." And I know it works because I've been doing it. But it's so at odds with the culture in companies, or what's the current way that companies are being organized that I don't think this stuff can work in most of the companies in the area. It could work on 20% of the companies that will be nice. And maybe if you lifted up, then it might be interesting for about half of the companies, but probably shouldn't even try it.

And to my big surprise, almost every company has tried to adopt agile and unfortunately many of them have failed. So I'm not sure if that proves my statement like, we shouldn't try it, but I think this is not something that can become common. It's still the thing that really, if you want Agile to work, you really need kind of culture, really the kind of mindset in there to take off.

Most of the companies unfortunately don't have this. So why even bother to try it? There's so much my failure in that. And the question is okay. And certainly, a lot of those failures by the way, are also called by the way that these agile initiatives have been done in companies. If you want to create self-organization and self-management in the company and you tell everybody what to do their work, that's not going to work either. So a lot of these agile initiatives are done in a non-agile way which already is a cause of failure. But even then am wondering, could this even succeed in the majority of companies?

Doug Talbot: I think that the manifesto people must truly be surprised that pretty much the entire world has tried this. And I think that we should be amazed that it's gone that far, where it's a common vocabulary in every organization on the planet. Whether they've tried it or not. And most of them have tried it in some part, in somewhere.

I think the thing that we possibly could have predicted is that if you put out something, that's a concept, a mindset, a philosophy, something quite amorphous, and then it's going to go in a million directions, and it has. And I think that the good news there is that some of those million directions are showing that they've got lifespan for the next 20 years. And yes, it's going to take long generational change times for everyone to grasp the concepts. And some people are going to go out of business, the Blockbuster effect, but in a process and culture sense, people won't realize that they're missing the wave and should be up and surfing by now.

Raf Gemmail: There was a Mary Poppendieck quote that I use for a short period, it was along the lines of, in any other industry, if something has been around for 20 years and it's still seen as modern and new, then there's a problem there, You're not having the growth. I worked on the waterfall projects. I think many of us did in that era. And I've seen the projects which were canned, which went over budget, where you worked on software and it never got released. And which went majorly over any estimates where the requirements were wrong.

Moving beyond the frameworks and brands towards pragmatic agility [49:03]

Raf Gemmail: Going through the journey of working with those teams as you're trying to do scrum, and then going through Kanban. That's like an evolution and seeing the DevOps scene come up. And I think there are values there persist. Like, if anyone asks me now, I'm like, "Forget the frameworks. Look at the values and principles." Because for me, they still hold, hugely.

Are we learning? Are we addressing the customer? Do we know what we're trying to solve here? Technical excellence. Isn't there for shiny, it's there because it helps us achieve a better product. How do we use those principles to govern our approach? And it becomes a lot more contextual, but I think there is still a lot to the early majority, the late majority. Scrum is the way to go.

People trying to hold on to the SAFe practices. Here's the framework. I myself remember trying to appease an architect at one stage by going into DAD and studying DAD and thinking, this is a nice compromise. But at the end of it, you realize you were just putting lots of constraints around a team's evolution. And I think the future is lean. It's moving. It's measuring, it's being empirical. It's improving continuously. Is my take on it. It's the DevOps. It's the DevSecOps. It's shifting left. It's seeing the whole value chain. It's optimizing flow.

So I think that approach has gone past the early adopters, perhaps to some parts of the early majority. Cause I hear, "We don't want to do scrum." And then you think, "Oh God, are you going to do waterfall?" But actually when you hear the language of people doing their own ad hoc thing, sometimes you pick up those lean principles in there. They're actually trying to move Quickly. They're trying to use design thinking. They're trying to use other approaches, which are very safe.

One thing I love doing my students is putting the design thinking graph up, the Agile principles, I think I used modern Agile, I put out the figure of eight, the DevOps lifecycle. There's a lean UX series of things. And they're all the same, right? They're about delivering value, optimizing the whole, continuously improving, learning, having learning feedback loops.

I think there's an essence there. Agile as a term was great. It flipped our minds and there are tools we can pick and choose from.

Craig Smith: I think the thing is, if you look at the 12th principle of agility, it's around continuous improvement. And so to me, I think they succinctly put in words, whilst we can debate some of those things. There's a few things that are a little bit dated and some emissions, but for the most part, that's the problem that we need to solve. And the 12th principle says we need to continuously improve. And so the Agile that I was doing 20 years ago, back in the early days when the only book I had was eXtreme Programming Explained, and a couple of papers.

I had dragged off this thing that was called the internet that I hit on a 56K modem. Now 20 years later, if you think about where we are, we needed something different. And so the things we can all bounce back to, and yes, I think we can be negative and look at things and go, "Yes, things aren't where they need to be." But that's also continuous improvement.

And that hopefully is going to push us harder in the next 20, to build the better frameworks or to build the better approaches.

We built DevOps because we, all of a sudden went, "We've got a cloud and we need to do something." And that's grown in ... Look at how that's grown in 10 years. It didn't exist 10 years ago. I think when we look at it at a much wider scale, do we have a lot to do? Yes, we've got wicked problems to solve and that's the call to action, I have for us in our community is, we now need to rally around those wicked problems.

Solve the business agility that we were talking about before, solve these issues. Those things are open on the table for someone to come up with real solutions to, because if we're failing, it's just giving us that feedback to go, "Okay, this isn't as easy as we thought and we need to look forward." So I'm optimistic for the future. I think we've shown that it's a little slower than we thought. Well, I mean, agility here try going faster. I'd like it to be a bit faster, but we have moved a long way in 20 years.

We’ve come a long way in 20 years, and there is still a long way to go [52:23]

Shane Hastie: And I think that's a great note for us to end on. We've moved a long way in 20 years. There's a long way to go and the reflection, learning and adapting. And if there's anything that we as the culture and methods crew at, InfoQ do try and do is provide people with ideas to reflect and learn and adapt from. So gentlemen, thank you very much for your time. It's been an absolute pleasure.