We recently learned that the paper I wrote, “Mylar [aka Mylyn]: a degree-of-interest model for integrated development environment (IDEs),” won the Modularity conference’s award for most influential paper from 10 years ago. Back then, Gail Murphy, Tasktop co-founder and Chief Scientist, and I set out on a journey to connect developers to the software lifecycle. Looking back, I am fascinated by the trend of improvements to developer productivity to date. Even more fascinating is the fact that we’re still in the early stages of this journey.
The Modularity Conference community is near-and-dear to me, and has considerable overlaps with the early members of the InfoQ community. Modularity is where crosscutting program structure ideas were developed, which eventually saw broad popularity through their adoption by Spring and similar frameworks. Crosscutting structure accepts that whatever dominant hierarchy users define for their software (e.g., the type hierarchy), they will end up with crosscutting concerns, such as security. Once developers start thinking in terms of crosscutting structure, it’s hard to stop. More than a decade ago, as a tool builder and developer working on the first Aspect-Oriented Programming tools, I saw an opportunity to capture the crosscutting state that emerges from how developers work and collaborate. Then came the epiphany. No matter how many millions of lines of code are in the codebase, for any given user story we’re trying to implement, or defect that we’re trying to fix, we only care about the slice of the current or future code that’s related to that particular unit of user value.
Crosscutting Modularity Experiment
The experiment of applying crosscutting modularity to ease the developer’s day worked. We managed to tie together Agile planning, issue tracking, source control, build and continuous integration artifacts, and make them a first-class part of the IDE, tied together with source code via “task context.” Mylar got renamed Mylyn because of a trademark concern, and was broadly adopted via its deep integration into Eclipse. Its core concepts shaped the Agile/ALM functionality of other IDEs such as VisualStudio. But that wasn’t enough.
The problem is that coding is only one slice of software delivery lifecycle. As I started working with larger development organizations, I learned that Agile planning, support, operations, UI design and product management all seem to get in the way of what developers love to do most—code. I learned that as passionate as I was to transform developer productivity via the IDE, developer tools alone would not be sufficient to achieve a 10x improvement because of the way that large-scale software is built. Solving software delivery productivity problems involves looking beyond the box of the developer and towards the end-to-end flow of value from great new business idea to delighted end user. When this connection is made, developers are able to spend the majority of their time delivering value through coding. Yet during the past 10 years we’ve seen tools become even more siloed, with seemingly each stakeholder using a different SaaS tool particularly tuned to his or her discipline. As inspiration on how to connect the application lifecycle with the trending growth in tool heterogeneity, I turned again to the principles set out by the Modularity community.
Modern Software Lifecycle
In a modern software lifecycle, the currency of delivering user value is represented by artifacts that reside in separate tool repositories tailored to particular stages of the lifecycle. This ranges from requirements management tools used by business analysts to service desk tools used by support personnel, and everything in between. Each of these repositories has a hierarchy of artifacts. Stand far enough away, and it looks just like a large program hierarchy and call structure. So the problem of connecting the lifecycle can be reduced to a problem of creating a crosscutting structure that connects the hierarchies of artifacts in each of those repositories. Requirements, user stories, support cases and defects all crosscut the various repositories and tools in which we separate and capture the context related to each of them. Each is related to the business value that we deliver.
Delivering business value connects the efforts of professional software practitioners. This business value is captured by the crosscutting artifacts, such as defects and user stories, that we create as we plan, wireframe, code and improve operational performance. Taking all of this together, the biggest thing that I learned from the past 10 years of working on this problem is that we are connected by the artifacts that we create. A user story connects a product manager to a developer. A support case connects a user to a tester and a developer. A requirement or epic connects the business strategy to the work delivered by a team for a release. The traditional view of Agile has emphasized individuals and interactions over processes and tools. However, as organizations try to scale Agile, we have learned that when interpreted to the extreme, this view is broken. At large scale, thousands of individuals are creating millions of artifacts using dozens of tools. As such, we now need to look at large scale Agile as individuals and interactions leveraging just enough process across integrated tools.
Looking at the problem from this point of view, there are three categories that each software delivery organization needs to define:
- The artifacts and process model for each stage of delivery
- The tools used to managed those artifacts
- How those artifacts will be connected across tools and processes
The successes and failures from the last decade of Agile, ALM and DevOps deployments provide us with a foundation for (1) and (2). Agile has come of age. Ten years ago there was still a lot of experimentation ranging from pair programming to Scrum. We now have hardened disciplines that span the range of organizational sizes. For example, early stage projects can thrive with Kanban, and sets of teams can be managed effectively with Scrum. In addition, enterprise software delivery can benefit tremendously from Scaled Agile Framework (SAFe), Disciplined Agile Delivery (DAD), Scrum.org’s Nexus or LeSS. Startups can connect their service desks directly to development, while enterprises benefit from modern ITIL deployments augmented by Application Performance Monitoring (APM). And in each of these categories, we now have a huge selection of great tools to choose from, each geared to different organizational sizes and process models.
While tools and processes have changed the last 10 years, the people who build software have not. The maturation of the processes and tools may seem to be at odds with this if developers have been asked to log into five different tools a day to update the status of what they are working on and communicate with colleagues in other departments. And this makes the next step in transforming software delivery very clear. Given that the software value stream is connected by the artifacts that we create, we need to come up with a crosscutting model of lifecycle artifacts.
Connecting development across best-of-breed tools and processes
To summarize, over the past decade the key thing I’ve learned is that in software delivery there is no single source of artifacts. Cool new developer tools, many of them open source, are created monthly. Developers jump on those great new tools hoping for a productivity gain, while other less technology savvy parts of the business continue to use the entrenched tools they have grown comfortable with. This trends is actually reducing, not increasing, the productivity of large-scale software delivery. We need the best of both worlds—both developer productivity and collaboration and integration across the large-scale software lifecycle. We’ve learned that processes should be independent. Regardless of the software delivery process -- Agile or other methodology -- processes need to work across a variety of tools and be implemented on top of those tools.
When you take these things into account, you can implement your Agile and DevOps transformation in a way that crosscuts the entire organization rather than limiting it to developers alone. Having enough of these processes in place makes all of the stakeholders involved in software delivery happier and more productive. You take a process and implement it across a variety of tools and artifacts in a way that keeps people connected. We need to break through those tool and process silos in order for the next generation of software lifecycle implementation to become more about collaboration and communication -- anchored by the code and other artifacts that we are creating in the process.
With these lessons in mind, to get started scaling your Agile & DevOps transformation consider the following steps:
- Define your end-to-end software lifecycle architecture. This can take a DevOps centric view if you are focused on continuous integration and deployment. Or a more Agile centric point of view if you are focused on the project and team side of things. Either way, create a diagram that highlights how value flows from business idea to functioning software, and all of the stakeholders involved. At this stage, ignore the specific tools and focus on the roles, teams and interactions.
- Define the tools that will support those teams and interactions. The reality is that each one of the stakeholders will live in their tool of choice, be that an issue track and source repository like JIRA and Git, or a large-scale Agile project and portfolio tracking tool like Rally, or an IT/Ops tracking system like ServiceNow. Expand your diagram from (1) to include the concrete tools that you have, but keep the first diagram around as the tools will evolve over time. At this phase you’ll realize if you’re missing a tool that’s needed to support a particular interaction. For example, you may note that your build tool is insufficient to support your vision of continuous integration and may decide that it’s a priority to deploy a more modern tool in this category.
- Define the connections between the teams and tools. Today’s tools are very siloed, as they work best when everyone in the organization uses that single tool. However, unless you are working at a smaller scale and only ended up with one box in step (2), you need to connect the tools to support the people and interactions that need to plan tools. Once you’ve identified those, determine which integration tool you’ll use to connect the tools.
All of the most successful Agile and DevOps transformations that I’ve seen have taken this approach to simplifying the problem of supporting people and interactions at large-scale.
Conclusion
The Modularity award recognizes the utility of creating a crosscutting structure that connects people through the software artifacts that they create. The next step in the evolution of the software lifecycle and developer productivity is all about building out this crosscutting structure a first-class part of our software delivery tool chain. In doing so, we will create the technical infrastructure needed to leverage the benefits of Agile and DevOps as we scale to the dozens and hundreds of teams that are needed to build today’s most complex and interesting software.
About the Author
Dr. Kersten is the CEO of Tasktop Technologies, creator and leader of the Eclipse Mylyn open source project, and inventor of the task-focused interface. His goal is to create the collaborative infrastructure to connect knowledge workers in the new world of software delivery. At Tasktop,Mik drives Tasktop’s strategic direction, key partnerships, and culture of customer-focused innovation. Prior to Tasktop, Mik launched a series of open source tools that changed the way software developers collaborate. As a research scientist at Xerox PARC, he created the first aspect-oriented development tools for AspectJ. He then created the task-focused interface during his PhD thesis and validated it with the release of Mylyn, now downloaded 2 million times per month. Building on the success of Mylyn, Mik created the Tasktop Dev and Sync product lines.