Why Traditional Test-Automation Tools Stifle Agility

In recent times, much excitement has circulated about the direction of next generation functional testing tools. Alas, many agile organizations still struggle to make their traditional record-and-playback automated testing tools work for them. Elisabeth Hendrickson, aka "test Obsessed", tells them why to stop.

Hendrickson nicely summarizes her message in the following, index-card-able way:

Why Traditional, Record-and-Playback, Heavyweight, Commercial Test Automation Solutions Are Not Agile
Three key reasons:

1. The test-last workflow encouraged by such tools is all wrong for Agile teams.
2. The unmaintainable scripts created with such tools become an impediment to change.
3. Such specialized tools create a need for Test Automation Specialists and thus foster silos.

After first describing how the test-last aspect of record-and-playback tools has little chance of success on any project, agile or not, Hendrickson explains why this is particularly detrimental to an agile project. On an agile project, test-last workflow has at least the following problems:

• Waste: the same information is duplicated in both the manual and automated regression tests. Actually, it’s duplicated elsewhere too. But for now, let’s just focus on the duplication in the manual and automated tests.
• Feedback Latency: the bulk of the testing in this workflow is manual, and that means it takes days or weeks to discover the effect of a given change. If we’re working in 4 week sprints, waiting 3 - 4 weeks for regression test results just does not work.

...Further, test-last tools cannot support Acceptance Test Driven Development (ATDD). Agile teams need tools that support starting the test automation effort immediately, using a test-first approach.

Hendrickson explains how the test scripts fundamental to these record-and-playback tools, which inevitably contain a spaghetti mix of business-level expectations and implementation-specific details about the UI code, turn an agile project's responsiveness to change into a maintenance nightmare. Succinctly stated:

Agile teams need tools that separate the essence of the test from the implementation details. Such a separation is a hallmark of good design and increases maintainability.

Thirdly, due largely to their high costs of and propriety-coding requirements, typical record-and-playback tools lead most organizations to the creation of a dedicated group of "Test Automation Specialists", charged as the keepers of the automated tests. Hendrickson asserts how this works against the collaboration required for effective agility:

Agile teams increase their effectiveness and efficiency by breaking down silos, not by creating test automation superheroes. That means the test automation effort becomes a collaboration. Business stakeholders, analysts, and black box testers contribute tests expressed in an automatable form (e.g. a Fit table) while the programmers write the code to hook the tests up to the implementation.

Hendrickson finishes off with a great discussion about what Agile teams do need from their test automation tools:

Agile teams need test automation tools/frameworks that:

• Support starting the test automation effort immediately, using a test-first approach.
• Separate the essence of the test from the implementation details.
• Support and encourage good programming practices for the code portion of the test automation.
• Support writing test automation code using real languages, with real IDEs.
• Foster collaboration.

Fit, FitNesse, and related tools do just that.

Do take a moment to read Elisebeth Hendrickson's full post for more on her insight and experiences. Also, see Brian Marrick's blog for more expert advice on agile testing.