The Agile Alliance Functional Testing Tools (AA-FTT) workshop was held on the day before the Agile 2012 conference in Dallas, Texas. Run as an open space, the session was open to anyone interested in talking about the future of functional testing tools and beyond.
Approximately thirty testing tool developers and users attended the event and discussed a number of topics that were proposed and facilitated by the attendees. The open space was facilitated by Matt Barcomb.
George Dinwiddie led the first session on "Enabling Non-Programmers" which turned into a lively debate with Corey Haines on whether non-programmers should actually be writing automation. George argued that he is keen for business people to be able to read and understand the tests and for non-programmers to be able to write them. As a result, he was looking for a way to refactor Cucumber step definitions into another step definition (as opposed to rewriting them in Ruby). Corey argued that testers who could not write simple Ruby scripts had no place on his team (and suspected that this view was shared by many in the Cucumber community). Elisabeth Hendrickson added to the debate that maybe Cucumber is not the right tool for the job and perhaps a tool like Robot Framework would be more aligned to non-programmers as it is more keyword-driven (she also suggested that it can be a double-edged sword because you can also make a mess if you don't give your tests some thought).
This discussion also covered the issues that many people are having in making an agile transition. Woody Zuill commented that he enables non-programmers by getting them to come work with his team every day. Christian Hassa noted that there is a difference between specification and testing and this is a fundamental shift for traditional testers. Corey Haines added that many of our problems are not our toolsets but that our training is about thinking in a procedural way rather than an example way of thinking.
Joesph Wilk led a session on "Not Testing", which is a practice that is gaining popularity in lean startups, where the emphasis is on hacking it out and releasing to production sooner, but taking a risk that sometimes things will fail. Joseph used the example of startups like Etsy and Scribd who deploy regularly but put effort into monitoring their systems in Production (and automatically rollback if necessary). Some of the discussion points included:
- the primary impact of success is the skill level of your developers
- calculating the cost of failure
- complexity is in the eye of the beholder
- quick delivery requires a quick feedback cycle
- need a small amount of effective tests of the things you are most scared of
- question everything
- this approach is geared to small startups
- some consideration is required on how to bring new developers on board and how to model for others to follow
Brandon Leiran was interested in the testing equivalent of coding katas, and the discussion identified a number of opportunities including:
- Weekend Testing
- Community Book Club
- Code Roasts
- Practice at driving different interfaces using tools like the Triangle Tester Exercise and the Parking Lot Calculator
- Test Case / Specification writing sessions
- Simulations
- Add tests to open source projects
Jeff Morgan was looking for some discussion around holes that participants were experiencing in current frameworks. As an example, he has been working on a Ruby gem to make the HTML reporting out of Cucumber better. Elisabeth Hendrickson was very keen to see some better reporting around high level feature coverage. She was, however, glad to see that tool developers were no longer focusing on editors and some discussion focused on the good support that editors like TextMate, Eclipse, Visual Studio and RubyMine have for testing tools like Cucumber. Another frustration was around test data management, extending on existing tools like FactoryGirl and Faker.
Finally, Elisabeth Hendrickson led a discussion around the future of the AA-FTT and her desire to hand over the group to the future group of leaders. Joseph Wilk has now taken over the leadership and will be looking to the community for support and how to best move the group forward.
More details on the AA-FTT, including outcomes from the session, can be found on their discussion group. Further notes and pictures from the session have also been blogged by the author.
