Since my old days at Xerox PARC I have always been sitting between basically application life cycle management and development tools and my emphasis was really about development tools at the start making new modularity mechanisms, making batter tools support for showing the structure of the system until I realized that there is this big gap between how we plan software and how we manage defects and issues and how we actually build, the day to day, how developers code. The really interesting the thing that’s happened in the last decade of planning what we deliver to costumers is Agile.
And it works; we’ve been using it on every project we got involved with ever since ’99 when Kent Becks’ book came out, but there is still this big disconnect between this sort of planning process and the actual development that happens on the ground. And I think a really important part of that conversation about Agile, the conversation that people are having here, that costumers trying to adopt Agile on a large scale are having, is how to connect those development tools to be Agile planning processes. And that is what Tasktop has done with this creation that we call Tasktop interface which basically links in the artifacts that you talk about when you talk about value to the user, about product features and then the code that is actually being written in a Java editor, in a C# editor or in PHP.
So we just want to be part of this conversation because we see a lot of potential for tools to really automate that loop between production and planning.
That is right. On the management team we just announced that we’ve got Neelan Choksi who is the former COO of SpringSource and more recently it was acquired by VMware with one of the largest open source technology acquisitions. He was also with the CEO of Lexcycle and that company created Stanza that was acquired by Amazon, he was also behind Solarmetric as well as CEO there and will bring him on a chief operating officer and also the president of the company, I am going to retain my role as CEO driving the vision and the execution.
He got together with Erich Gamma and they made a tool that enabled XP with JUnit. So I honestly find a bit of irony with this sort of opposition to tools and then the need for them, the fact that there is this concrete need. It’s great not to distribute a team, you should try in every way not to do it but then for a lot of adopters of Agile it’s impossible. As soon as you're distributing your team, you either need a camera pointing at a physical bit of paper, or you fall back into an email exchange about what priorities that should be on, what is actually on the cards. And I think the backlash against high ceremony, these tools that were around at the time the Agile manifesto was written was a real backlash, it made a lot of sense.
There were a lot of vendors selling snake oil, there were a lot of tools that really enforced process so rigorously they didn’t allow people to be creative. And to code the way that they naturally code and to communicate that they naturally communicate. So I think it’s just time for this next generation of low ceremony very Lean tools and I think that is where we are getting to today. And JUnit is an example of that. That was a very simple tool, very special purposed, very lean and it fit perfectly into XP.
Yes, definitely. The reason everything that we’ve done to date and we are expanding on this as you are allude to, but a lot of what we’ve done today has been built on Eclipse, because its goal was to become a tool platform and IBM initially and now a large ecosystem of companies did a tremendous job, making a very extensible tool platform. In the same way that Emacs was just so nice and extensible for macros and for textual interaction, Eclipse became extensible for both textual interaction but also for these structure views, these trees, these ways of navigating very large code bases. So it became a really good platform for application development tools.
And by virtue of doing that it became so extensible that we were able to take this application life cycle management process, this Agile process, this notion of the fact that developers don’t just work on source code, they work on tasks that then change source code, they work on a user story or they fix a defect. They were able to layer in that on top of Eclipse IDE without having to rewrite the whole platform. So it became a really good platform for us to create this innovation of the task focused interface and to bring that life cycle management process right into IDE. So that is why it has always been the platform for all our experimentation. And there has been a lot of interest now in bringing some of these technologies to Visual Studio to other tools.
For those developers we’ve created a standalone desktop application they can use right now, the Tasktop desktop application that runs on Windows and Linux and not too long from now, on Mac. So it’s just been a great platform to leverage to hook into the way that developers work, the developers’ day to day activity. And Mylyn is the way that we do that. Everything that we needed to add to the IDE to support Agile and application life cycle management tooling in the IDE is Mylyn. It’s basically all these hooks for making sure that you can bring in tasks, sources, issues, builds into the Eclipse IDE which originally was only architected around application development tools, and compilers and runtimes and debuggers.
5. Did you actually start the Mylyn project?
Yes I did. Around 2003 this disconnect between the planning loop and the software production seemed fundamental enough for me that I just decided to ditch industry for a while and go do my PhD to figure out what was going on, because I didn’t understand the problem to this level at that point. I just noticed I was wasting a ton of time, I was always going to the same information, I was getting RSI from all the mouse clicks was doing, I was sort of introspecting both my own work and then the great thing about going to do some research was I could get to watch other people work.
I could watch developers at IBM, watch how they program and sort of undid some of my own biases with having worked with their teams in the past. And I know this that everyone was always just going back to the same class, to the same method, going back to the same information over and over and they this constantly a really part of the day, looking for that same information they needed to get work done. And if we could somehow chunk those things that they were always referring to and hook them on to something rather than having them look at tens of thousands or hundred of thousands of lines of source code, that’s in the system they're working on, then we could stop that. We could bring up everything that is related to what they are doing at this point, have them shelve that and come back to it later.
And the experiment was to say: "Ok, let’s not make files be the primary part of the development environment, let’s make it be task, let’s make it be user stories, defects, issues, cases, whatever you have. And that is what we did. And so you interact with the task, you say: "I want you on the user story now" and then everything you touch, everything you work on, everything that is related structurally to that is all that you see in the IDE. And you can switch tasks with a click and that was the whole foundation for Mylyn, all that core framework is in Mylyn.
That’s what Tasktop builds on with its integrations with commercial ALM systems, commercial Agile product tracking tools and now we are going higher up the stack with today’s announcement of our partnership with Accept Software. We are now integrating with product and the portfolio management tools as well.
Because Eclipse has this steep learning curve, there is no question of it, it’s a problem, we’ve seen the way students adopt it, we’ve seen the way people from other backgrounds and text editors adopt it and there is this steep learning curve at the start because there is so much, hopefully what is essentially complex in there. There is so much to learn about the way that debuggers interact, there is just a lot of tool support in there. Also because of the extensibility of Eclipse, its strength and the reason it won is that it can be extended to new languages, for example. It’s a little bit more generic than the special purpose IDEs, like Xcode which is all about creating pretty GUIs with a little bit of Objective-C in there.
So what we made sure to do is to simplify the developers’ interaction. So Eclipse does not look any simpler, but we make sure that you only had one pane and that’s your tasks and that is everything that you work on and then make sure that you can access that from the IDE. So what happens is that when the developer is not used to Eclipse, but they are using some kind of issue tracker, let’s say, they are using a Bugzilla or Quality Center. They naturally work on defects, they now see those inside Eclipse and they could activate that defect and they can click a stack trace and then you go straight to the line of Java code you know that was from the defect.
So rather than your interacting being going to the web browser and selecting the class from the top of the stack trace based on that and figuring out where the search functionality is, all those sorts of things, we were able to simplify that away. So we can’t simplify that look of Eclipse, but we do simplify the interaction that you have by just reducing the amount of searching and clicking you need to do. So in that sense it is easier for a newcomer to adopt, but you still have to learn how the compiler works, how to set up your projects and those sorts of things.
8. So your sympathies are with the programmers or are they with the Scrum managers?
My passion, my nature is in programming, I don’t do as much of it anymore and unfortunately I thing I am no going to do as much as I’d like, but now I think you have in order to get any sort of new way of working, of planning, of doing anything adopted in the software engineering organization, you have to focus on the programmers and I think that is exactly what’s been lacking, they've been an afterthought. There’s been all this innovation, these very nice web user interfaces and tools have been created for helping project managers keep track of what is going on. Now what is going on you’ve got a user story and that is great, that is captured, it’s not in MS Project anymore which is nice because you’ve got some Agile process and tooling and UI to work with it.
But if what is happening is that the programmer breaks down the user story at the end of iteration or at the end of the release into subtasks because you told them to do it, it was not a natural thing for them do to it, it did not save them any time to break it down for themselves as they were working, you lose it. The things that you are after as a manager, having actuals tracked, knowing what work is going on, if there is an impediment because a subtask of this user story was figuring out the 3rd party API you needed and the developer got stuck. At that point you just have no idea. So I think it’s actually very dangerous the fact that programmers have been left behind to a degree in the last few years of Agile tool creation, because at very large companies, at very large levels of Agile we see this exact problem.
Those costumers come to us and say: "Developers are not adopting us". They are being told to 2 months after they took their Scrum training course, they are all very excited about this because there are some great ideas in there, but when it comes time to the release crunch it just goes out the window. The user stories that came from management, whether it’s a product owner or something else and then they're not broken up and the activity, Agile becomes a documentation technology for figuring out what happened, not actually have developers break things up as they go and collaborate around the backlog. So that is really what we are trying hard to address, that is all that Mylyn is about is bringing those concepts from planning into a form that developers can use day to day.
So let’s say that you are actually working on a user story, you’ve got it active, you see only the code that is related to that, you the program fail, the web application fails, it dumps log; you can in your IDE select that log, right-click it, go "New defect" and that gets added to your user’s story. So it’s 2 clicks and after those two clicks you’ll see all the code relevant to that defect because we can analyze that stack trace and say: "Ok, this is the context of the new defect". So the developer finally has a reason to do that, to actually decompose things down to the level that management wants them to.
And yes, we're seeing developers doing that, there is more work that we need to do, just to always streamline to make it smoother because and to giving developers more incentive to decompose things that way, so you do get the visibility. But I don’t think Agile is going to deliver on its returns unless we go back to the programmers.
This is a very interesting thing and it goes back even outside of the software development. There is this movement of Taylorism in production where you would just watch workers at everything they are doing so you could optimize your production process of some sort. I think if you do that, if you pull the big brother move on developers, you hired them because they are smart which means that they are smart enough to game their behavior, so that you see that you want to see. They’ll create subtasks at the end of the release, they will do it very quickly and it will look like that is what they’ve worked on. So I think what we are trying to is give developers the technology for micromanaging themselves and have that aggregate up into something that management needs.
So for example I used to have to fill out DARPA time sheets when I worked on Aspect Oriented Programming at Xerox PARC. And every Friday I would make up what I spent my time on because in software engineering we multitask all the time trying to figure out how many hours working on something, it’s almost impossible. But the tool that we give developers with Tasktop is every Friday, if they are doing time sheets, you just see everything you spent your time on, you can always adjust your activity, you manager doesn’t see it because we consider that to be private, anything that is tracked is always private, but then you can push it into your project management tool, after you have adjusted it.
So you say: "Ok, over these 4 days, you see this little chart, I’ve spend 2 hours here, 1 hour here working on this particular user story. I actually spend 2 hours talking to someone about it in the afternoon and I wasn’t active on my computer, so add 2 hours there and then that’s it." You push into into any of these management tools. But what we do is with anything that is automatically tracked, we always give the control to the developer. So they can always adjust it, they can change what was tracked; they can edit their context so that we don’t get into this big brother mentality where we adjust people’s behavior because they think they are being watched. And it’s working.
Developers were the ones that asked us for this feature, you always have pressure for developers for more automation and pressure for management for more transparency and visibility and especially in software I don’t think you can do the thing where you are watching every click, it’s just counter productive.
I think a lot has been learned these last 10 years, both on the side of the different methodologies, the popularity of Scrum. The fact that some of these vocabularies work really well for people. On the next 10 years I think like with objects you get from the early adopters, you have to go to pragmatists, OOPLSA got for me really boring, when it’s 5 of the 15 papers in the track I’m looking at are about optimizing bytecodes. It stopped being so conceptual, it got very pragmatic, but that’s a really important thing to have that pragmatic dialog for people writing the VMs, writing the debuggers and the rest. So I think something similar could happen to Agile, if Dave is right.
This discussion has to turn to pragmatic issues with like for example scaling and training. I think the thing that’s near and dear to me it’s capturing everything that’s commanded dialogue for the last 10 years and baking that into the tools. So if you pick a methodology, you can capture that in the tools, you don’t have to train everyone and then retrain them to make sure they follow methodology, you can capture it, you have to make it lightweight enough. I think we’ve learned that: we cannot unlearn that lesson, it has to be JUnit not any of these very large testing frameworks and tools like the ones that have not been successful. I think it’s great that we learned that.
I think it’s going to be more pragmatic and I think the tool support for what we’ve learned in the last 10 years is going to become more key in the sense that it’s great what has happened for managers, but we now have to bring that down into the way that developers can consume and directly benefit from in their day-to-day work, not just in long term that the release has more user value, but actually makes the developer's day-to-day work easier because that work is not that easy given the size that systems have grown to in the last decade.
11. You personally have any glimmers of what you are going to do at post ALM?
I think there is this cycle of technology moving from software development, because some of the smartest people in software development tend to want to solve their own problems, so they are going to build their own environment, they’ll build SmallTalk, which will then make a desktop, Lisp which made InterLisp and then became a desktop. Desktop environments form that way. I think a lot of what we learned is going to start applying, a lot of what we’ve created in the Agile community is going to start applying to other niches. Scrum is probably not the best way to define your marketing department, but a lot of the ideas of Lean planning and having just-in-time information, having a tight feedback loop, I think do apply.
I think some of these tools are going to go beyond software, like a natural step from ALM to product management, PPM and then to planning, it's going to be time to displace this sort of Microsoft project style of managing organizations and teams across industries into this more Agile and Lean style. I guess we would make a 360 degrees back to the production process that where it came from, if that happens.
I think it's a really exciting time. We’ve got with these releases we’ve got today. It’s great what we’re able to do. It was first of all capture these ideas of the Agile community, make them work inside the IDE, across vendors - that’s really exciting that’s happening. The fact that we’ve got this competition between the Agile vendors, which means there has been a lot of innovation and which also means that you get stuck into these vendor silos where tools don’t talk to each other properly. The baking that we’ve done is to demonstrate that the tools can work across the silos. We made this de facto framework called Mylyn, it’s open source, most of the Agile vendors build on it now and they can provide these consistent things like time tracking, task focusing, like the new Tasktop which works across all the Agile vendors tools.
I think that open source taking a more leading role in ALM is going to be very interesting, get distributed version control, I think that naturally map into and Agile and Lean production process. They’ve got this notion of tasks, they work very nicely with Mylyn. It’s pretty exciting both from the point of view of what the vendor are doing, what’s happening in open source and then of this integration framework that we have Eclipse and Mylyn and that Tasktop is building on.