Cédric Beust

TestNG started as an experimental toy project. The main motivation was that despite using JUnit for years, there were still a lot of concepts and approaches used by JUnit that I didn't feel comfortable with. The more I investigated, and the more I disagreed with some of its core philosophical design decisions, such as reinstantiating your class before each method invocation (thus forcing me to use statics if I want to maintain state between invocations) or not allowing me to create dependent tests, which are crucial for functional testing.

Overall, I just found that JUnit was doing great in the area of unit testing but was lacking in other places required by any kind of functional or acceptance testing.

Back then, I was also involved in the annotations JSR and it occurred to me that annotations would be a good fit to mark test methods.

Finally, the last piece in the puzzle came from a web site with the strange name of del.icio.us that introduced this bizarre new concept of "tags". It occurred to me that it would be neat if I could tag my test methods and then be able to invoke any arbitrary group of them without having to recompile anything. I renamed these tags "groups" and I realized that they would be a great match for annotations as well.

With these ideas in mind, I set out to create TestNG 1.0 almost four years ago and I published it just for fun and to get some feedback.

What did you want to accomplish in writing Next Generation Java Testing?

To my surprise, TestNG was extremely well received and despite the de facto monopoly that JUnit has on the Java testing world, a fairly large community embraced it and started taking it toward all sorts of interesting directions. In the first few months, I could barely keep up with the list of feature requests. It was as if there was a big contingent of Java developers who had been resigned to using JUnit for all these years and who suddenly saw a way to participate in the creation of the next generation of testing frameworks.

Luckily, I was quickly joined in the project by Alexandru Popescu and together, we started implementing and discussing all kinds of innovative ideas with TestNG users. TestNG as it is today was truly shaped by its users, and Alexandru and I have just been helping to make it happen.

The TestNG group has more than 1300 subscribers today, and for the past couple of years, the idea of trying to capture all the wisdom and interesting ideas that have been discussed on the mailing-list started to emerge. I was approached several times to write a book but I always bailed because I didn't feel I could find the time or the energy to undertake this endeavor.

I have to give credit to Hani for finally pushing me over the edge after he offered to write this book with me. Suddenly, the whole project looked a lot more feasible and for having known Hani for several years, I had absolute confidence that we could come up with valuable and innovative material that would truly help the Java testing community.

How was Hani to work with for readers that may only know him from the BileBlog?

Hani is actually quite different in real life than he appears to the readers of his blog. A lot of people still don't understand that the Bile Blog is an act. Hani created an online personality that allows him to publish his thoughts in a totally uncensored way, and there are still quite a few people who just can't see past the language.

But anyone who does immediately sees that there is very concrete and first-hand experience behind every single post that he writes. There is nothing gratuitous or ill-informed in what he writes since he only discusses topics and frameworks that he has personally used (and most of the time, suffered through).

I recognized this quality fairly quickly in his posts and a lot of his conclusions resonated with me, so like a lot of people, I got interested in meeting him. He turned out to be very different than I expected. He's mild-mannered, soft-spoken and remarkably polite, but as soon as you ask him about technical matters, his words are extremely sharp and straight to the point and surprisingly devoid of any cursing... They also reveal a very deep and extensive knowledge of Java and associated areas, a very critical and inquisitive mind coupled with first-hand real experience of how very big software systems work.

Hani and I are pragmatists: we like to consider problems objectively and try to solve them after weighing every single option without any personal bias, and we've tried very hard to convey this objectivity in every single chapter of our book (with the exception of the one called "Digressions" where we allowed ourselves to discuss certain topics a little more freely. BileBlog fans will probably love this chapter more than the others :-)).

What do you think the most common mistake developers make in respect to testing code?

Well, the very first mistake they usually make is not to think about testing when they start writing code. No matter how proficient you are at writing tests, if the code you want to test wasn't written with testability in mind, your work is going to be severely crippled unless you start engaging in serious refactoring (which is not always an option).

If you are in the enviable position of being in charge of both the code and its tests, your life is going to be made easier if you start wondering early how you can make your code testable. Note that a lot of people interpret this to mean that test-driven development is the only sane way to program, and Hani and I strongly disagree with this form of extremism. TDD can be useful, but there are other simple principles that you can keep in mind while writing your tests last and still reach a clean and entirely testable piece of code.

To whet your readers' appetite, let me throw in a few thoughts on how to make your code more testable:

• Avoid statics. It's unfortunate that singletons and similar patterns have made statics so popular because the world is now plagued with a lot of code that is harder to test. One way to address this problem is to use a dependency injection framework (Hani and I are big fans of Guice, which we cover in our book, but Spring is also a good option).
• Don't hesitate to modify your code to make it more testable. It if means making a private method protected or public, it's usually a good idea to go ahead because the immediate gain of enabling automatic testability will usually outweigh the potential maintenance burden that you've just added upon yourself by making this method more visible.
• Interfaces are not as bad as a lot of people want you to think. A lot of XP developers will tell you not to introduce an interface unless you're sure that there will be at least two implementations, but our position is that every single concrete class can potentially be used in two different ways: in production and in test. Extracting an interface is usually a move that makes your design cleaner and more testable.

Of course, we discuss these issues and many others in depth in the book.

What is the most difficult type of code to test in your opinion?

The first two examples that come to mind are graphic interfaces and mobile software.

Historically, graphic and mobile interfaces have always been hard to test because their API's were never designed with automated testability in mind. Things have vastly improved in the graphic world (see for example FEST if you need to test Swing or SWT code) but they are still fairly abysmal in Java ME and Java mobile in general. The main problem in the mobile area is that the emulators are black boxes that cannot be driven from outside, thereby making it very hard to do functional testing on running applications, and also that the MIDP and Java ME class hierarchies have been extremely poorly designed and are not showing any signs of improving with the laters JSR's.

Hopefully, the project that I'm currently working on (Google Android) is solving some of these problems and we did our best to make testability and sane design at the forefront of our thoughts. Please let me know if you disagree :-)

Testing code running in application servers is also far from trivial, but we have made a lot of progress in this area lately and we now have a good grasp on how to achieve it thanks to the multitude of frameworks that were created these past five years just for that purpose.

Where do you see testing and TestNG going in the future?

I think the main progress in testing was achieved these past few years, during which more and more developers realized that making automated testing possible is part of their job. It wasn't too long ago that the idea of writing tests was scorned by most developers, who considered it as not worthy of their time and talent. Ten years ago, testing was just a task that was delegated to an obscure QA team that developers would barely interact with. Going back further, I find it really interesting that most of my CS teachers would never even bother writing tests. We would turn in projects and programs and they would have a piece of paper to verify that we covered all the cases correctly.

Things have changed radically these past years, and the only thing I can see changing in the future is that submitting code without tests will be seen in as bad a light as submitting code that doesn't compile.

Overall, I believe most modern languages (Java, C#, Ruby, not sure about Visual Basic, though :-)) have the testing part covered. The frameworks are there, usually fairly flexible, but what we are still missing is tools. We've made a lot of progress these past years, but we're still very short of environments that would tell you right away that the code you just wrote is incorrect because it's breaking tests or that would analyze your code and suggest ways in which it can be improved and tested.

I think the most interesting advances in testing in the next few years will come from tools, and I can't wait to use them.

Hani Suleiman

What did you want to accomplish in writing Next Generation Java Testing?

The one thing that struck me time and time again when it came to testing is how little of it I actually saw in a professional context. The majority of unit tests I came across tended to be in open source projects, and the disconnect has always bothered me.

Over time, I had sort of resigned myself to the fact that testing doesn't really lend itself to the kind of work that I do (EE stuff, across the board). That these systems are too complex to test easily, and that its an unfortunate fact of life.

Along came TestNG, and I started talking to Cedric about it. The one theme that resonated with me is that it seems to place a very strong bias towards enabling testing, than forcing me to test in any particular way. Dependencies were nothing to be ashamed of, stateful tests are sensible and in fact sometimes the right way forward, and other such principles than flew in the face of (at the time) conventional wisdom.

Along the way, this nebulous concept of 'testability' started to gel. There are pattens that emerge from testing code, and when trying to see what others are doing, there seemed to be very little in the way of concrete real world examples or insight. The examples out there are usually quite trivial, and don't really show how they'd apply to real problems.

I had had numerous conversations with many testing people, and was often hugely impressed with the techniques and designs that people have come up with quietly as part of their jobs.

However, all this seemed to be locked into the minds of a select few. There's a huge disparity between those who think that testing is important (almost everyone) and those who successfully apply it as part of their development process, with very little to encourage and guide the former to become the latter. Thus, this book was born!

What do you think the most common mistake developers make in respect to testing code?

The assumption that all tests should be trivial and quick. This is true for a very small subset of software. It's fine if tests take a long time to run, it's fine if you need a real database, and it's fine to have stateful tests that depend on other tests.

What is the most difficult type of code to test in your opinion?

Along with Cedric's thoughts, I'd like to add database testing. Yes, there are a bunch of tools to help you do it, and a variety of techniques to help alleviate some of the problems. I haven't heard yet of a good solution of writing tests against an existing set of moving data. If you go against a known static set, you slowly deviate from the live db. If you use live dumps, you have to deal with security issues and ensuring you're not exposing sensitive information to developers, and so on. In the banking world the production world is heavily walled off, so it becomes tricky ensuring that a test database is meaningful and current and can be used concurrently with automated tests.

Many developers write tests but don't worry about "coverage". Why do you think coverage is essential?

Well, there's also the corollary to that, which is that developers who do worry about coverage, tend to obsess about it! Coverage is useful purely as a rough guide as to what you're not testing. The actual percentage of coverage is almost completely irrelevant. The book discusses the evils of the green/red bars we've all come to know and love, and how important it is to resist the temptation to make the red bar smaller and smaller.

Code coverage is useful in that it shows you that your tests do what you think they do. It's vital to keep in mind that coverage reports for example have nothing to say about missing functionality, since the code isn't there to be checked in the first place! It is just another item in our testing toolbox that can used for good or evil. Evil usage would be to diligently ensure all your code is covered (even to the point of adding comments in your source code to tell the coverage tool to ignore certain blocks!) Good usage would be using it to verify tests and to try and spot areas that need further investigation. The book goes into detail as to the appropriate usage of code coverage tools.

You wrote the chapter on JEE testing including topics such as JPA, JDBC, and JNDI. Many developers I know find these tests difficult from a setup and scope perspective. How do you balance writing end to end tests that require a full container and database (but replicate the full environment) versus lighter tests that make use of mocks for key JEE constructs?

I'm fairly strongly against the use of mocks for Java EE constructs. These APIs are often complicated and come with whole swathes of tests to verify compliance. End users never see what a pain in the ass it actually is to certify a product as compatible to a given EE API. Mock implementations on the other hand are most certainly not certified. The danger of using them is that over time, people start implementing more and more of the API, its more code that can go wrong, and more code thats written just for the sake of making your tests look good. Increasingly, it becomes more and more divorced from the real implementation.

My personal opinion is that refactoring is a far superior tool to test such code. The book works through an example of modifying a login servlet through refactoring in order to make it testable.

Having said that, there is a time and place for mocks and stub objects. If the API is trivial enough, or if you have a third party dependency that you can't refactor away, by all means, mock away!

In the integration chapter you discuss Swing UI and Selenium testing. How have you been successful with such UI driven testing on your projects? Some developers argue that the time required to write such tests is not worth the limited number of bugs that are found as a result.

I'll be honest and say that the answer is...no. I grudingly agree with developers who say that its not worth the effort. Testing pixel perfect placement is something that as of right now, is best done by real people. Things are a little different though when we get into the RIA world and ajax apps. In these cases, there is often a lot of logic in javascript, and so its important to apply testing here too. Often the exact same principles that encourage us to write testable code in Java apply to javascript. For example, not mixing presentation with business logic, having isolated components that can independency wired up, and so on. Tests for such applications can be browser based, with a number of js test files.