Jaroslav Tulach's latest book Practical API Design covers the topic of API design of software projects. Jaroslav discusses the importance of API design in the modern software applications, what are the different factors that make a good API, and how to go about implementing API frameworks. He brings his experience as the architect for NetBeans IDE project to the writing of this book. In the book, Jaroslav talks about several real-world examples of how to (and more importantly how not to) use Java API based on his experiences working on NetBeans project.
InfoQ spoke with Jaroslav about his new book, the main motivation for writing it and other topics. Topics covered in the interview include the assessment and validation of the quality of software, role of agile and lean software development methodologies in the design API frameworks and architecture and design reviews.
We are also making an excerpt (~4 MB PDF) from the Practical API Design book available for our readers.
InfoQ: What was the motivation behind writing the "Practical API Design" book?
Jaroslav Tulach (JT): The book is based on the notes I've been collecting over the past ten years while designing and maintaining the NetBeans APIs. Also, they're based on the process of transferring the NetBeans API knowledge to the rest of the NetBeans development team. I started to think about converting my notes into a book about five years ago. However, it took a bit of time before I really got started. At various points, other tasks got in the way. Other times I was simply afraid of not being able to finish the book, as well as the fear of being rejected by publishers, etc.
However, in the summer of 2007, I chatted to my wife's cousin at a family party. When I told him of my doubts, he said: "You know what to write about; you know it is an interesting topic; you know you are an expert in it. So... why haven't you started writing yet?".
I was always reminded of this comment whenever my motivation to finish the "Practical API Design" book flagged, which was to be expected in this kind of long running project. By the way, let me take this opportunity to thank everyone again who helped me finish my book.
InfoQ: In the book, you discussed the "Component Injection" technique to achieve a Modular Architecture. What role design concepts like Dependency Injection (DI), Aspect-Oriented Programming (AOP), and Annotations can play in software development in general and API design in particular?
JT: I believe that modularizing applications into distinct parts is highly desirable. It's even better when those pieces aren't aware of each other, so that someone can assemble an application from the various independent parts. To allow this kind of assembly from loosely coupled pieces, it is essential to have some form of injection. Dependency injection in the Spring style is fine. java.util.ServiceLoader works as well. I dedicated chapter 7 of my book to a comparison of those and other alternatives.
There used to be a time when people went nuts on hearing the phrase "bytecode manipulation". That's no longer the case. People aren't afraid to execute bytecode different to that produced by the Java compiler anymore. However, if you tell them to go directly into the .class file and manipulate the bits, the fear returns. Why is that? I believe this is due to AOP. AOP make bytecode changes normal, without requiring understanding of the class file format. In fact, AOP can be seen as a high level language for bytecode manipulation. It is not as powerful as making changes to the .class file directly, yet it is approachable to the masses and generally understandable. This perfectly illustrates the principle that good abstractions make everything more usable. The principle is true in the world of bytecode manipulation as well as in that of API design.
We started making heavy use of annotations during the development of the next version of NetBeans IDE. Basically, we are defining new annotations as a facade over our old XML-based API. During compilation the annotations are processed by our annotation processors. At the end of the processing, the correct (old and complex) XML is generated. I cannot begin to describe how happy we are with this solution! Our APIs are suddenly nicer. Registration information is now part of the Java source code. Code completion automatically works in IDEs. All registrations are verified for correctness during compilation. Based on these experiences, I can only recommend compile-time annotation to all API designers!
InfoQ: You wrote in the book about how to check the quality of an API library. Can you elaborate on how the teams can have an on-going assessment and validation of the quality of software being written?
JT: Common wisdom among programmers is that design cannot be done by committee. However, how can we design systems of increasing size without designing in teams? Doesn't working in a team hurt consistency? Yes, partly. Most people seem to be able to keep design consistency when they work alone, although software projects of today and of the future are designed in teams. Keeping consistency in such environments is much harder, but is certainly possible.
As usual, one has two choices: either detect regressions in quality when they happen or prevent such problems from occurring before they are integrated. I have dedicated the complete chapter 16 to these topics. There I describe what aspects need to be verified and checked and how to check them automatically. In this way, we can be prepared for when things start going bad. Also, the chapter introduces an "API Review" process, which the NetBeans team is following to review API changes before integration. This way we can prevent things to even think about regressing.
InfoQ: How can Agile and lean software development methodologies like SCRUM, XP and Kanban help a project team involved in the design and development of API frameworks?
JT: The question is whether agile methodologies help design and development of API frameworks... or whether properly modularizing an application and splitting it into many libraries with APIs simplifies the use of agile methodologies? Both are probably true.
There are several sound rules in the world of API design. For example, you need to be aware at the outset that the first version of your API is not going to be perfect. Also, you need to envision who your users are. Two other basic principles are that unit test coverage is almost a must and that you should always design your API in such a way that it is ready to be evolved further.
Many of these pieces of advice are close to being the governing rules of agile methodologies as well. As such, I think that proper API design and agile methodologies can only strengthen each other.
InfoQ: Can you discuss how your team conducted architecture and design reviews in NetBeans development project?
JT: We have two modes: the standard review and the fast-track review. The latter is used for small, incremental, compatible, and non-controversial changes. It is based on the "optimistic locking strategy": you prepare the code change diff and attach it to the issue tracking system. People then have a week to comment or veto the change. If nobody objects in that time frame, the change can immediately be applied. This works well for single method changes or class extensions to already existing libraries.
The standard review is targeted at reviewing a whole new library or subsystem. It is a two-round review. Firstly, we review the concept. Then, if the concept is accepted, the result is reviewed again before integration into the main code repository. All the review details are documented on NetBeans website.
InfoQ: What are the best practices and "gotchas" that software architects should keep in mind when working on creating reusable component libraries?
JT: It is difficult to enumerate them all... it took me 400 pages in the Practical API Design book. :-) But let's pick up on something interesting: "What do you think is encompassed by the letters API?" The names of classes? Probably. The names of their fields or methods? Yes, if they are public or protected. But is that all?
No, it isn't. Did you ever think of the files your application reads as being an API? And aren't the open sockets also an API? What about the environment variables? And the localized messages? How about the text output produced by your code? All these can influence the behavior of your application or library. They can also be observed and used externally. As such, it is therefore some kind of API. It is very important to realize that and to always keep it in mind.
InfoQ: What is the role of software architects in the current economic and market conditions?
JT: Tough question. The general answer is to learn to offer what the market needs. But I suspect nobody is really sure what that is. Except...
...one of the problems in software development is associated with "big bang" changes, that is, situations where you find that your product has too many bugs and design problems, to the extent that it is unfixable, requiring a complete rewrite. Coming to this point takes a very long time, with the evidence usually needing to grow over several releases. However, there is a point in the lifecycle of most software projects where someone offers to perform a "bigbang" change. Initially, the offer is declined because everyone understands the rewrite will be painful and costly. Yet, over the next releases, while the evidence mounts, the team realizes there is no escape. At that point, the whole project is stopped, an incompatible rewrite of a subsystem is forced, and then the other teams need to adapt to the change. In spite of how realistic the original plans were, this process usually takes much longer than expected. And, in the end, we now have a shiny new product which is less buggy than before but handles only about half the functionality of the original. Needless to say, this is not an effective process at all.
The Practical API Design book describes best practices to avoid the need for these "big bang" changes. First of all it explains what compatibility is and advocates small, incremental, backwardly compatible changes, while always focusing on future evolution. A mode of this kind does not need disruptive "big bang" changes at all. On the other hand, sometimes such large changes are necessary. That is why the book discusses ways to provide alternative co-existent behaviors and explains how to bridge them, when necessary.
If we modularize our applications and treat each piece as a library with an API, we can minimize the need for stop-the-world "big bang" rewrites and replace them with continuous distributed improvements. This is maybe slightly more expensive at first, but the long term cost of ownership is definitely lower.
All that said, leads to the following answer to your original question: "Maybe software architects should learn more about proper API design and use it to create more cost effective product teams."
InfoQ: What do you think about the new features and APIs in the upcoming JDK Version 7 and the dropped features like Closures?
JT: The #1 thing that Java needs is a standard module system adopted by everyone. Recently I argued with a Ruby guy about the benefits of API design in Ruby. Originally, it seemed that "duck typing" is Ruby's big advantage. Then we agreed that Ruby's real advantage is all its "gems" with all their library dependencies... while Java has nothing. This needs to be fixed.
I know there are existing module systems. By the way, one of these runs underneath NetBeans IDE. However, by no means every library in the world is designed with modularity in mind. This has got to be changed. There needs to be support for modularity in the Java language itself. I'm even willing to wait a few years for closures, if I can get the modularity in Java right now.
InfoQ: If you have to pick a feature that you like the most in Java language, what would it be? And the feature that you least like about?
JT: Java is said to have many weaknesses. However, I guess one of its weaknesses is also its biggest strength: Java is said to be verbose. True, you can write shorter programs doing the same thing in many other languages. So, Java is verbose. However, the benefit of verbosity is that people are also able to read what they've written. Java programs can be read and understood by maintainers of its code. Moreover, this can be done on a printed page, without needing "go to declaration" functionality and the "code completion" feature of your an IDE. I guess I'm not really satisfied with Java's verbosity, however I do appreciate the readability that flows from it.
InfoQ: Lastly, other than your own book, do you have an IT and a non-IT book recommendation for our readers?
JT: My book references Effective Java and Gang of Four's Design Patterns book. Those are a must read and indeed they are well known.
Beyond these I also reference a perfect book by Petr Vopenka about history of geometry: "The Key Stone of European Knowledge". It may sound boring at first, but it is not and I felt in love with it. Many of Practical API Design's "philosophical" sections are based on ideas of that book. So if you belong to the camp that finds those parts interesting, I'd recommend the book to your attention. There is just one problem: It has not yet been translated to English.
InfoQ: Thank you Jaroslav.