The Divisive Effect of Separate Issue Tracking Tools

Separate issue tracking systems for Development and IT Operations are a source of conflict and ineffectiveness for many organizations. For effective change management, in particular database changes, we need shared issue tracking systems where Operations teams can see upcoming work from Development and Development teams can see details of live service issues logged from Production.

Overview

Teams evolving data-driven software systems can benefit from looking afresh at the way in which issues and work items are tracked in order to be as effective as possible.

A tooling boundary acts as a wall between teams, which is why teams using different tools to track work or defects often have difficulty working together. Instead, both Development and IT Operations teams (including database administrators – DBAs) should generally use the same tool or at least be able to find crosscutting information through a common search interface.

A crosscutting example is production incidents originating in code or architecture flaws. If these incidents are stored in an issue tracking tool accessible by Operations only, then Development will miss the context and history behind them. It's not uncommon that Development provides a fix that turns out incomplete because they were only told “what” failed, not the “why” and “how”.

Another example are planned database changes. If this kind of work items are stored in an agile board or a project management tool that is not accessible or known by Operations, then they will not be aware of the incoming change. The end result is often rushed deployment of database changes that do not meet operational criteria due to pressure from product/marketing teams.

In short, issue and work item tracking systems should have a low barrier to entry, ideally with a browser-based interface along with an API for custom integration and reporting. The browser interface improves accessibility and ease of use while an API helps integrate issue tracking in the delivery lifecycle, for example a developer might be able to set an issue as “Fixed” via a commit comment with a predefined pattern such as “Fixes issue #4”.

Avoid 'one tool per department'

Many organisations miss the opportunity to bring together Development and DBA/Ops teams by allowing each department to choose its own issue and/or work tracking tool: the Development team usually chooses a lightweight tool that integrates with version control but that has little support for Production incidents, while the Ops/DBA team tends to choose a tool that excels at service management but has little or no support for version control integration or agile/iterative software development. The result is that knowledge is held in silos, and cooperation between DBAs/Ops people and Developers is difficult.

Instead of separate knowledge silos, aim for task and issue tracking that is at least searchable through a common portal or interface, allowing Developers to see incidents from Production and DBAs to see details of forthcoming database changes currently in active development.

Whichever tool you choose to use, make sure that knowledge and awareness of issues are not kept in separate silos, and that the way in which the tool is configured helps with collaboration and communication between different groups.

More specifically, breaking down silos requires communication. Alas, tools only store information, they don't communicate them. This means you need some kind of (lightweight) process to identify and exchange the relevant bits of information from Dev to Ops and vice-versa.

Searching

An issue tracking system might have dozens or even hundreds of open tickets. Closed tickets might be on the order of thousands per year. How can one make sense of which tickets are relevant?

Ops to Dev

Let's see how an Ops incident can be searchable by Dev.

Typically an Ops team deals with incidents across multiple systems or services, each assigned to a development team. So first of all, issues need to indicate which systems were affected during the incident.

Still, you might have dozens of incidents where the same system was involved. One possible way to break this down is to include custom fields that identify which components of the system were directly affected (database, backend, frontend, etc.). Another, more straightforward possibility, is a custom field “Review by development” to be checked by the issue submitter (or, if possible, anyone with access to the issue).

Or, in case the Ops issue generated one or more bug fixes for Dev, the original issue could simply link to its “children” Dev issues. The problem here is that there could be issues that either don't require code changes at all (but are still relevant to Dev, for e.g. Ops included a new log from the system in a centralized log search such as ELK) or require a large change that cannot be described by a single issue (for e.g. re-architecting the application to improve poor performance under certain load).

Dev to Ops

In waterfall development tasks such as creating a new database or adding new customer fields were common. However, in today's agile methods work is more commonly divided into user stories or features, spanning all the layers of the stack (“vertical slicing”).

That makes it hard for DBAs for example to understand if a story will require database changes and to what extent.

The tool you choose should allow an easy way to identify which application components are involved in the story, either via labels, custom fields or, even better, integration with your source control system (for example, automatically linking to commits whose comment references the story).

Another option, similar to the one in “Ops to Dev” is for example to label (or identify in any other way) a story with non-trivial database changes as “Review by DBA” (tip: definition of trivial database change needs to be agreed by both Dev and DBA).

Communicating

So the information is stored and it's searchable and easy to identify for Dev and Ops. This doesn't necessarily mean teams will start taking action.

Furthermore, the time required to review issues is not negligible. Telling the DBAs to regularly review Dev items that affect the database probably won't be welcomed with open arms, when they're firefighting problems in production on a regular basis.

Because we're trying to break down silos by providing timely feedback to each other, a simple way is to start promoting regular meetings between Dev and Ops (possibly alternating the exact participants). This helps ensure people dedicate some time in their agenda for this. It also creates room for blameless communication between the teams, as they identify areas of improvement out of the pressure-ridden deployment or incident workflow.

Be careful, however, not to turn this into a change approval board, the goal here is to promote interactions between divided teams, not adopt new formalities for getting changes deployed!

Previous preparation by both team members is highly beneficial, at least initially. Each team's homework includes a list of issues the other team should review, in decreasing order of importance. For each issue clarify why it's important, what changes are planned or implemented, which systems they affect, etc.

We also recommend to time box both the meetings themselves and the time spent on each issue, to avoid time drain and detailed discussions of an issue (schedule time after the meeting when an issue warrants it).

Finally, make sure you're addressing not only the issues themselves and improving on those, but you're also reflecting on the review process itself. Is it being effective? Are you uncovering problems earlier? Are teams becoming more aware of each other's workflow and impact on the other? Are you spending too much or too little time reviewing issues? How to improve that?

Once you feel this process has matured, you might want to look into partially automating it (if all teams involved agree it's a good idea). For example, the DBA's could be notified of changes made in certain database definition files or Dev could be notified of issues that affected the system(s) they are working on or maintaining.

Tool Recommendations

One of the best tools to use for issue tracking with modern software delivery approaches like Continuous Delivery is Atlassian JIRA. Not only does JIRA have excellent third-party integrations and programmability via an API, but JIRA also makes it easy to configure different workflows for different teams - Agile, Kanban, Service Desk – so avoiding the very common trap of 'one tool per department'.

JIRA's Kanban view for Development teams

JIRA's Service Desk view for IT Ops teams

By making use of JIRA's integration between Service Desk and developer-focused ticket workflows, Development and Ops/DBA teams are empowered to collaborate on investigation and fixing of problems, and improving the software system in general through access to a rich store of relevant details.

Choose an issue tracker that is suitable for both Developers and DBAs/Ops

The issue tracker you choose should be friendly and accessible to both Development teams and Ops/DBA teams. Many tools that provide an issue tracking feature are designed primarily for service desks within IT Operations teams, and are therefore not suitable for use as issue trackers for software delivery. Service desks are often set up to deal with support calls from a single individual rather than teams, and default ticket access permissions tend to hide ticket details from everyone except the person who first happened to report the problem.

This means that the selection of an issue tracker needs to be a joint endeavour between software development and IT operations, including the DBA team. Tickets need to be visible and shareable between Dev and Ops/DBA teams, and ticket workflows need to be configurable and flexible; at a minimum, tickets must be cross-linkable between Dev-focused areas of the tracker and Ops-focused areas, even if their workflows are different.

When you must keep multiple tools

As previously mentioned, a single tool that supports both Development and Operation workflows and issue types is ideal to reduce friction and promote reusability and shared understanding.

However, there are situations when a migration to a single tool is not possible. One clear example is when outsourcing either development or operations. It might be impossible to have the outsourcer use the same tools as the client, as the former can't be forced to switch tooling for each client.

In this case it becomes essential that tools on both sides support an API-driven integration, with strong access control (such as LDAP authentication across both organizations for both tracking systems). You will either need to create queries on both systems to search for relevant tickets (see section Searching for details)  – hopefully already built-in the tool – or develop a custom CRUD integration that synchronizes between the tools (added benefit here is only one tool usage required in each team).

Whatever the solution, this option requires more work and is more error prone, thus not recommended unless there is no other choice.

Make all tickets visible to all people and teams

Everyone involved in building and operating the software systems should be able to search and view every issue. At least all people in the technology department should have access, perhaps not to modify, but certainly to comment on all tickets. Specifically, do not use issue trackers in a way that allows only the reporter of the issue to see the details.

Often the main problem resides in the aggregation of different types of tickets (internal service desk, customer-facing software systems, etc.) in a single tracking system, with a single user access control. The security needs of a fraction of tickets (for example, providing a new iPhone to the CEO) should not override the need for transparency and sharing between teams.

This implies that 'service desk' setups (dealing with requests related to employee devices like printers, desktops, mobile phones, etc., or creating accounts and changing access rights) should be separate from issue trackers for core software systems.

When we build software, the quality of the software – and therefore the issues that we track – are the responsibility of not just a single team, but of multiple teams, often in different buildings, countries, or even organisations.

That split is the first step on a route to unifying the ticketing for the business systems. The “service desk” team (or role inside the Operations team) can keep their own tracking system, while new tickets for business systems go into the new multi-purpose (Dev and Ops) tool.

Conclusion

The choice of issue tracking tools and usage might seem as a no-brainer: pick the tool that does the job best for the team. But in today's world of continuous service delivery and operation, teams need to work closer together than ever.

Letting each team independently choose their tool of preference is a form of local optimization that leads to unintentional information hiding and keeps teams further apart.

Of course, the tooling of choice needs to be able to support the workflow of all the teams that rely on them. But they also need to support tracking of cross-team relevant issues, via ease of use, searchability and extensibility/integration with other tools. These characteristics are a foundation on which to build strong and early feedback loops both within and between Development and Operations.

For extensive coverage of collaboration issues between Development and Operations, particular DBAs, see this series of articles on Database Lifecycle Management

About the Authors

Matthew Skelton has been building, deploying, and operating commercial software systems since 1998. Co-founder and Principal Consultant at Skelton Thatcher Consulting, he specialises in helping organisations to adopt and sustain good practices for building and operating software systems: Continuous Delivery, DevOps, aspects of ITIL, and software operability. Matthew curates the well-known DevOps team topologies patterns and is co-author of the books Database Lifecycle Management (Redgate) and Continuous Delivery with Windows and .NET (O’Reilly). @matthewpskelton

Manuel Pais is a Dev+Build+QA = DevOps advocate (and InfoQ lead editor). People-friendly technologist at Skelton Thatcher Consulting. Jack of all trades, master of continuous improvement, Manuel enjoys helping organizations adopt DevOps, Continuous Delivery and Cloud, from both technical and human perspectives. He has worked on a range of technologies (Java, .Net, web, mobile) and industries (banking, telecom, defense and aviation). @manupaisable 
Quick facts: favourite race: fim da Europa (end of Europe), favourite wine: red, favourite accomplishment: being a dad.