A Transformation Journey for a Distributed Development Organization

Agile transformations are never easy, but they are even more challenging than usual when it comes to geographically distributed teams. Any major enterprises must take such a challenging path, because of diverse organizational structures, responsibility sharing, utilizing off-shore/near-shore models, and many other reasons.

Konica Minolta’s Workplace Hub (WPH) program is no exception — we are using geographically distributed teams to develop and deliver. Ours is an ongoing journey, during which we have changed many processes to provide effective solutions for our teams and our customers. It’s also important to look at the broader team: stakeholders. They are not part of the development team, but agile is about bringing people who are relevant together, to work as one single unit. Such a scenario is common in large enterprises.

I will highlight some of our experiences helping others who are taking the same journey. I’ll share some of the methods that worked for us during our journey, as well as those that didn’t. This story is about the organization and processes, but most importantly, about the people and the mindset that helped us succeed..

Undertaking a Complex Project: Workplace Hub

The Workplace Hub (WPH) project is complex, both technologically and organizationally. From a technology standpoint, it includes a server placed at customer premises, as well as components that reside in data centers (DC). The server includes custom HW components, third-party SW components, as well as custom-developed services. All these components on the server side and the DC side are connected, to make support operations possible. This setup creates a very complicated ecosystem from a development perspective, although the customer value is exactly the opposite — all this complexity is isolated from them.

Many technologies and components used in the WPH ecosystem are the newest ones, that make use of the latest domain advances. The numerous hardware and software components deliver value and simplicity to the customer, and provide enough tools to the support team to proactively monitor and manage issues that may happen on the WPH and support the customer when necessary. Protecting customer data and keeping the system available on the customer’s premises also creates additional complexity that increases the number of components. The integration use cases between these components are quite complicated, and to make them manageable, the team has to follow and rely on the latest technologies.

Organizational complexity happens for similar reasons. Because the product brings together different roles and departments (such as sales, support, and development), in many cases emerging requirements have to be analyzed, agreed upon, and prioritized before the implementation, and confirmed by different roles during and after implementation. The project team is also quite international, which adds to the complexity. Finding common time zones, ensuring the availability of the interested parties, and getting the necessary agreements can sometimes be challenging. So, we use technological solutions to bring people together virtually, when being together physically is not possible.

Our Vision and Goals: Why This Transformation Journey?

Our vision is multi-fold. The first step is to become the best software development team in the enterprise and to be recognized as such. We want universal understanding and an unwritten agreement with internal stakeholders that our team is using the best and most-efficient approach for our customers. Ultimately, we want to act as role models for other teams, share how we work, and replicate the model elsewhere. The next step is to be recognized as a respectable, visionary software development company in the industry.

Konica Minolta, a manufacturing company for over 145 years, has a historically close relationship with waterfall methodologies. But, this methodology doesn’t really allow fast customer value delivery, or direction changes based on customer feedback.

Just like many other modern organizations, we need to deliver quickly. But, we also need to adapt quickly as the customer and market demands change due to technology or business. This need means we don’t have one set of final requirements from the beginning—the trademark of waterfall. Instead, we deliver incrementally, adding features on top gradually, and removing or modifying features based on customer feedback as we go. This approach means we define only a small number of requirements at a time, and work on them iteratively.

This vision dictates an Agile organization and setup. We’re aware there is no end to bringing better and faster customer value. This is an ongoing journey, where we strive for perfection, sometimes taking major steps, but mostly changing minor things.Those minor things add up and create a difference.

One of the biggest challenges in our Agile organization is reusing some of the existing skill sets in teams, while hiring new teams in different locations for the new domains. This approach allows us to use the strengths of the company, but brings a challenge of creating one big team with a single goal. This also makes our experience stronger and more educational for other setups inside and outside the company. Basically, we reuse our existing multi-functional printer (MFP) related technical skills, as well as sales and support skills, and add newly hired teams for new functionalities that do not have any counterpart in the enterprise. This setup is not uncommon for large enterprises, and it provides extra motivation to be the best team and overcome the challenge of being distributed, by utilizing teams’ skills within our organization.

As mentioned earlier, we use some off-shore partners to work on our development. This arrangement adds to the distributed organization and presents challenges to organizing how work is done. We want these teams to use the same development principles and integrate seamlessly into our delivery pipeline.

I should also mention that this project is Konica Minolta’s first global experience with Agile—another reason we want to act as role models within our company and spread knowledge.

Our Approach to the Transformation

We’ve taken a two-pronged approach — big and small steps at the same time. Some of the bigger things we have done are described below.

We streamlined our organization by removing many management roles. We had a very deep hierarchy in our development organization, which made decision-making very slow and information flow very difficult. Removing some layers of management and channeling the former managers to customer value-adding work improved both situations dramatically.

At the beginning, we had:

• software-delivery managers, responsible for sets of components
• the teams working on these components

Both the managers and the team reported to:

• the next management level, whose purpose was to oversee the development.
• this next management level, in turn reported to a program manager
• more levels went up from there

This setup created too much reporting to upper levels,which is against Agile principles, as progress has to be seen by joining the demos anyway. It also resulted in no autonomy for the teams, because all the decisions were made by the upper echelons. Management discussed and prioritized  the work to be done, and the priorities were very unpredictable. This made life very difficult for delivery teams, because they did not understand why priorities changed, what the customer value was, what had to be done, or how and when, etc.

To fix this issue, we empowered the teams together with the Scaled Agile Framework (SAFe) transformation, by which we started applying SAFe related roles and processes in the organization, channeling the managers into customer value-adding domains or transferring some of them to different projects where their skills were more needed. This was not easily accomplished. We did our best to communicate why this change was necessary, what we were trying to do by empowering the teams, and how we could utilize their skills in other project domains or within the company. In the end, we managed to strike a deal with most teams.

Value Stream Teams

We created value stream teams, like the User Access/Management domain that delivers user access-related functionalities for the platform. Each team owns a domain and delivers value by working end-to-end on that domain. This approach allows each team to see a customer viewpoint in their domain. Before establishing these value streams, teams were component-based and some had no customer interaction, so they didn’t have any knowledge of the end result of their work. For example, one team was responsible for developing the authentication mechanisms and another was responsible for the identity provider, and they were completely dependent on each other, unable to deliver any value on their own. There were many similar dependencies and conflicting priorities.

With the new value stream model, the teams can work directly with customers, better understand priorities or requirements, and work with much fewer dependencies on other teams. This is  an ongoing process, because establishing a value stream team by itself  is sometimes not enough for team members to start taking action and be responsible for customer satisfaction. We continue to improve in this area.

What Worked

The changes that clearly worked and made teams happy — reduced dependencies, clear visibility into the customer benefits, being able to prepare their own plans (instead of being directed by management), and following them. Risk resolution and escalation and customer contact has, so far, had varied success and is an ongoing journey. But, in general, all teams confirmed that the transformation of the team structure was a success.

Outsourcing companies have used the same approach — working as value stream teams and delivering value to our customers in specific domains. They join the Program Increment (PI as defined in SAFe) events and the coordination and collaboration meetings, and work as an extension of our own teams. We’ve tried different approaches with some companies, but teams that are only responsible for components, don’t see the customer value, and there is too much back-and-forth and responsibilities are unclear.

Establish Metrics

Establish a set of metrics that are monitored by all levels. By using these metrics, you can see problems earlier and take proactive or reactive measures to sort out the issues. Our teams use these metrics to see and reduce their carry-over ratio, average bug resolution times, deliver on time, and track story points for capacity planning and delivery. These metrics are tracked by the teams for their own tracking purposes. There are also other metrics teams decide to track themselves, when they decide to improve some aspect of their tasks and see how they’re doing.

Create a Release Process

Releasing a version every three months has been a big accomplishment for us. We created a release process, and branching and repository structures to support this, and adjusted our development and QA processes to enable this. These changes enabled us to deliver faster customer value and have much less work-in-progress, which was previously creating many bugs because there was a huge amount of unreleased code. It will be an ongoing challenge to continue such improvements, and reach the "Release-on-Demand" model, after completing each feature.

Initially, releases treated the complete ecosystem as a bulk, running tests for the complete ecosystem and making sure everything worked and was releasing everything at once. This approach was obviously very waterfall and aligned with the traditional approach of the company. After many discussions, we broke the releases into separate streams, and also limited the amount of development before releasing each version. This separation enabled faster testing and faster delivery, but as mentioned before, the intention was to trim down this functionality delivery even further. We’re aware that actually speeding up releases means faster customer value and better quality, because less functionality is developed each time, which means less space for bugs.

It’s a challenge to convince different stakeholders such as QA, sales, and support, because some of them still follow the traditional mindset of "more testing = better quality," and don’t see the relationship between the amount developed for each version. This challenge is also highlighted by the DevOps reports we are using to convince our internal stakeholders.

Roll out the Scaled Agile Framework (SAFe)

Rolling out SAFe (Scaled Agile Framework) to the whole distributed development team was very useful for us because it clarified processes and job-role definitions. Before SAFe, there were different practices applied by different teams, and even the job-role descriptions were not universally understood. SAFe removed these obstacles for us — it defined the job roles for product owner, product manager (PO/PMs), release train engineer (RTE), scrum master (SM), and scrum team members. These roles had different meanings in different locations before, and we were able to consolidate them with SAFe.

SAFe also has enabled teams to prepare their own plan for a limited horizon during the Program Increment event. This was the biggest progress in our team empowerment—teams planning their own work and following their plan. Another big benefit of the PI is that it brings everyone together, to have open discussions and understand different viewpoints coming from different roles or teams. The sheer social benefit of being together in one place actually helps a lot.

Applying SAFe in a distributed environment is a common challenge. Here is how we do it:

• We hold the PI event, a key milestone for applying SAFe organized after every five sprints, in the location where most of our development teams are located, to require less travelling.
• Key members travel to the PI, to talk with other teams about dependencies, requirements, and planning.
• All the teams complete and present their plans during this event
• Plans are followed during the next implementation duration.

Regular Meetings

Regular meetings like Plan-Do-Check-Act (PDCA) or SoS Scrum of Scrums (SoS) are organized so distributed teams can talk to each other about delivery status, emerging risks, dependencies, breaking changes, etc. We should also mention that technology is our friend here — all teams use Jira and Confluence to follow plans and look at other pieces of work being done by different teams. They connect to others using calls, or direct messages via Slack. Slack communication is especially vital for us, because while it is asynchronous, it is 1-1 and the communication is usually very direct and to the point.

PI, regular calls, and Slack make SAFe work for us. It’s definitely not as efficient as being co-located, but it works and we are able to focus all the teams on a single purpose. Before SAFe and plan agreements, we were struggling to create this common purpose. Now, with the help of the PI and common presentations, everyone focuses on the same targets, after they talk to each other and agree on goals.

These are all big steps. But of course, many small improvements have been and are being made by the teams. Things like fine-tuning tools, attributes we are using, step-by-step improvements to our CI/CD pipeline, and increasing our automation are some examples.

The Challenges — How We Are Overcoming Them

We had many challenges. The biggest challenge was mindset. People are instinctively protective of their environment; most people don’t want to go out of their comfort zone. This was and will continue to be our main challenge. One of Agile’s key principles is being open and willing to change to deliver more and faster customer value. It’s all about constantly evaluating, questioning the current status, and asking, "What can be improved to make things even better?" Applying SAFe was actually one of the challenges, because everyone believed we were doing well at acting Agile, and that a new methodology would not bring any benefits. But, we started seeing the effects after the training sessions and the first PI events.

Similarly, many metrics were very difficult to establish, because collecting these metrics meant team members had to log some extra data, such as how many bugs developers found and how much time they spent solving bugs during the development cycle. The discussions and explanations help teams understand which metric can be used for which purpose, and how the teams use the metrics to improve their results. In general, collecting this data and continuously analyzing it brings many more benefits than the cost of the effort to collect it. Data collection methods are also openly discussed by the teams to find the easiest method. This is a challenge in terms of mindset, because collecting this data and having the team question itself about improvements is a transformation in itself. The team uses this info to improve itself. This kind of behaviour requires constant effort and focus, and that’s not easy.

Another challenge that is common in big, waterfall-oriented enterprises, is the clash with waterfall. Because waterfall has been used by the company and the members for a very long time, it has become a tradition and has made its way into all processes. Naturally, all company members are used to the methodology and the processes, because they are the standard, and they’ve applied them (many) times in the past. Establishing an Agile setup is always difficult in this kind of environment. Luckily, market demands and the support of some key managers allowed us to follow Agile practices. Because we want to deliver quick value to the customer, get our experiment results faster, don’t have the full set of requirements, and are developing a platform incrementally, waterfall is definitely not a fit. We now deliver releases much more frequently, we have a process to determine priorities and planning for short durations per SAFe, and we can demonstrate the benefits of Agile. All of these accomplishments would not have been possible with waterfall. But we have not yet overcome this challenge. When we want to take the next step with different projects or broader teams, we’ll need to leverage our success so far.

We also faced silo behaviour challenges because we had different teams in different countries. The physical distance and being apart created boundaries between the teams, and there was never enough collaboration between the geographies. Teams were not acting as one team focused on the same goal. Unfortunately, in many cases teams blamed other teams from other locations. This was mostly overcome with technology, more frequent and regular meetings, clear job role and responsibility definitions, and also by rolling out SAFe, which helped clarify processes and the collaboration model basics. We use all opportunities to bring people together and to create virtual teams to work on common issues.

Spending time together and working together solves many issues by itself. The PI event brings many people from different locations together and everyone can talk about their dependencies and plan together, and be exposed to each other. This helps create a common purpose for the teams. In cases where we need to do something internal or product-related where all teams participate, we select members from every team and let them act as virtual team members until the end of that activity. Another method we’ve found useful when a new process has to be rolled out or a big architectural change needs to be explained, we send team members to act as ambassadors to ease the roll-outs. Boundaries remain a constant challenge that needs to be consciously addressed by management everyday.

One of the bigger challenges we’ve faced is stripping down management levels, and making the organizational structure more Agile. This is also an ongoing challenge because we need to streamline the organization even further when it touches Workplace Hub in any way. Naturally, many people think they need to be part of the decision-making process, when in reality a single point of contact by the team can resolve many things in a much faster manner with the customer. As is written elsewhere ad infinitum, people, especially managers, cannot take their management hats off easily.

We removed unnecessary management layers utilizing SAFe and rolling out the RTE role. We used the PI events to let the teams carry out their own plan interacting with others (internal customers included), clarifying the stakeholders’ priorities (the team PO/PMs included) before the PI events so that the team could choose from the priorities. This showed managers that the teams were capable and even more efficient in coming up with their own plans, and their primary role was supporting them to address the bigger risks which teams could not resolve themselves. This eased our transformation and we managed to fill our missing roles with our managers, in some cases transferring them to different job roles within the company. We managed to convince most of the managers by showing our success and results.

It takes time to get used to new methods, so we allocated enough time to adjust, and also provided a lot of support for our people who were changing roles; they were naturally concerned about their job security and contribution levels. When we clarified they were still going to contribute and they were still very much needed, just in other fields, the conversations became easier. In some cases we transferred them to other projects, because that was what they preferred.

Accomplishments — From a Business Perspective

We are now releasing continuously and bringing more customer value with each release. This is  big progress for the business.

Our prioritization process has become much clearer. Previously, people contacted development teams and asked for something to be prioritized. There were almost always priority clashes with other people’s priorities, and on top of that every priority change was quite sudden and the teams had to change their focus. This naturally created a lot of technical debt, as well as frustration for the teams. We established a single process of prioritization, where all  stakeholders took part. This helped all the stakeholders openly discuss what was to be done and decide together which task(s) were higher priority. This brought everyone together and provided much clarity.

Feature implementation, developed for KM internal teams, is done by constantly being in touch with the users. This brings the double benefit of users being satisfied when they receive the update, as well as the development team ensuring they are doing the right thing for the customer. This approach has prevented many wasted hours and  created a much happier support team (the users of these functionalities), and will continue to do so.

What We Learned

This isn’t new, but it never hurts to repeat it: people are your most important asset! Hire and keep the best people on your team. Empower them to make decisions. If you have people who are motivated to change, and see things through,you’ll be able to transform and get results. The SAFe transformation gave us the infrastructure to empower our teams. Now, they can create their own plans and follow the plans using Scrum ceremonies.

There have been many other examples of (virtual) teams or members being empowered to take on projects, and bring solutions to the whole team. These projects’ output continues to be useful for all, and it also motivates the team members.

Another key learning point, especially with distributed teams, is that information sharing is very important. Key messages like vision, strategy, and goals should be repeated over and over again, in case someone has missed it (and believe me, someone will!). Key information should be placed in central locations where everyone has access. Calls remain the main tool; sharing information during regular calls has worked for us.

Bringing people together, face-to-face as much as possible, makes a big difference. It brings down cultural and national barriers faster and more easily. The PI event helps with that, but more solutions in addition to the PI must be considered by teams. Even though there is no measurable outcome, collaboration significantly improves if people come together from time to time.

Technology is our friend when it comes to getting people together. We organize regular calls in different contexts and on various topics. Using tools like Slack, Jira and Confluence helps a lot. Slack communication is especially important for us; it brings virtual teams together. We will be using webcams and TVs as much as possible to create "virtual gatherings."

Following an established methodology like SAFe helps. Team members can simply go online, search for keywords, and look for examples. This clarifies the answers to many questions. But, there is also a downside to this. Some members become too attached to the methodology and try to follow it to the letter—for example, reading the process on the SAFe website and saying, "This is not my job role’s responsibility" or "This is how it should be done." Actually, this should not be the intention in applying Agile methodologies; the real goal is to enable collaboration within the Scrum team as well as the broader team, to achieve results. As long as members can work collaboratively toward a single goal, the rules can be bent. As the first item on the Agile Manifesto states, "Individuals and interactions over processes and tools."

What’s Next in Our Journey

We have some major goals. For me, the most important goal is creating a culture that enables us to work on the transformation and to deliver results. All teams and companies have a culture, emergent or consciously established. We work to establish such a culture, which brings people together for a common purpose and helps them continuously transform. establishing core competencies and applying them during hiring, performance evaluations, and career development will help us see where this journey takes us. These core competencies will encourage Agile behaviour and foster collaboration. Depending on how we manage to develop these initiatives, this could be the next article’s topic!

Another big goal is to be able to release on demand. We’ve managed to make our releases much faster than before; but this is just a start. We want to be able to release in smaller increments and bring even faster customer value. We’re working on the infrastructure and processes to enable this. Our versioning and packaging should be configured to support this new schema. More importantly, our current QA process needs to be modified to test only updated or newly implemented functionalities, instead of the whole system. This approach will dramatically reduce the time we spend before each release.  To do this we also have to allow automated unit and integration tests. Another dimension — we’ll need to inform and rain our stakeholders working in the field to work on the new schema once we’ve defined it clearly.

We also want to focus on increasing our automation levels, so teams are able to work more on customer value — bringing work instead of doing reworks and tests. This will simplify our direction changes.

About the Author

Burak Ilter is the Head of Engineering at Konica Minolta. He is an IT professional with a long and diverse career. He’s worked for major companies in different roles, ranging from software engineering to system engineering, and from architecture to engineering management. He has practical experience with different programming languages, methodologies, and business domains, including the public sector, defense sector, finance, healthcare, and productization. He is married with two children. He enjoys reading science-fiction, history, and is interested in cycling and running. He is also an avid Japanese anime fan. He presented the topic of this article at the Lean Digital Summit 2019.