Layered Architecture for Test Automation

Abstract

In test automation, code involved in testing is not only test logic, but also a bunch of other supporting code, like url concatenation, html/xml parsing, UI accessing, etc. Test logic can be buried in this unrelated code, which has nothing to do with test logic itself, making test code hard to read and maintain. In this article, the layered architecture of test automation is presented to solve this problem. In this layered architecture, the test automation code is divided into three layers: (1) test cases, focusing on the test logic of the application, (2) the domain layer, modeling the system under test in domain terms, encapsulating http requests, browser control, result parsing logic and providing an interface for the test cases layer, (3) the system under test, which layer 2 will operate directly on.

Problem

QA's job includes designing test cases, exploratory testing, performing regression tests, etc. While some tasks like exploratory testing require intuition and smarts, some others, such as regression tests, are repetitive and laborious. As more features are added to the system, time consumed by regression tests gets longer and longer.

Test automation solves this problem. With test automation, repetitive work like regression tests is done by computer, and tests cases are translated to computer program, so that QA can be freed from the burden of routine test repetition to focus on more creative work.

In test automation, code involved in testing is not only test logic, but also a bunch of other supporting code, like url concatenation, html/xml parsing, UI accessing, etc. For example, to test a web service which carries out operations like search by different keywords and return an xml containing certain information (like customer information), the test automation code must:

1. Assemble a URL based on the operation under test,
2. Send out a http request with some http libraries,
3. Interpret the response sent back from the web server and parse the xml,
4. Compare results returned to expected results.

In some test automation code, all this URL concatenation, html/xml parsing, XPath expression and test logic code gets written together, usually in one class or one method.

This form is easy to pick up and is intuitive initially, but it has its problems:

1. Test logic is hard to understand and modify. When test logic is embedded into a large amount of other unrelated code, it's difficult to see what is tested. To add new test cases, one often has to reread the supporting code and find out where is the best point to add new code. Test logic becomes hard to understand too.
2. Tests become fragile. Since test logic and supporting code like html parsing are mixed together, one small change in the 'contract' between the system under test and the test automation code can break the test automation. For example, if the UI changes, like moving an input element to a different div tag, or changing an ID of some UI element, all test code operating this part of the UI is affected.
3. Maintenance cost is high. There are generally several test cases for a particular part of a system, and a large section of each test case is similar. For example, they may all have to (1) assemble a URL based on the operation under test, (2) send out a http request with some http libraries, (3) interpret the response sent back from the web server and parse the xml, (4) compare results returned to expected results. Since this code is duplicated in all of the test cases, if anything changes, you will need to modify each test case.

Solution

The domain of software development has experienced the same thing, and developed a solution, that is 'Layered Architecture'. Basically, the value of layered architecture, to quote Domain-Driven Design, is:

"The value of layers is that each specializes in a particular aspect of a computer program. This specialization allows more cohesive designs of each aspect, and it makes these designs much easier to interpret. Of course, it is vital to choose layers that isolate the most important cohesive design aspects."

Though the focus is different in the domain of test automation, the fundamental problem is the same, so a similar solution can be applied:

The test cases layer consists of multiple test cases. These test cases are based on the domain layer which encapsulating system under test in domain terms.
The Domain Layer accesses system under test directly.

Example

Say we are testing a restful web service. With this web service, you can search for some customer information, with telephone numbers as keyword.

To call this web service, the get http request in the following format should be sent out:

    http://{endpoint}/subscribers?telephoneNumber={telephoneNumber}
    

The piped data returned contains the subscriber's name, phone number, address, and other information:

    13120205504|ST|C|SQ|112|||FIRST|ST|W|Riverfront|BC|010|68930432|
    

Test cases for this service are (1) Search with a phone number which has an exact match, (2) Search with a phone number which has several exact matches, (3) Search with partial phone number.... The number of test cases is only limited only by the imagination of QA.

For each test case, the process is essentially the same: (1) assemble a URL containing the telephone number keyword, (2) send http get request with http library, (3) parse the piped data, (4) compare the data received with expected values. To avoid the problems mentioned before, we apply the layered architecture:

Test Cases Layer

The implementation of this layer is test framework related. In this example, we are using C# and NBehave (but not in a traditional way, please refer to the post 'Fix NBehave').

    [Story]
    public class SearchCustomerbyTelephoneNumberStory: TestBase
    {
        [Scenario]
        public void SearchWithAPhoneNumberWhichHasAnExactMatch()
        {
            story.WithScenario("Search with a phone number which has a exact match")
                .Given(AN_ACCOUNT_WITH_PHONE_NUMBER, "01068930432", EMPTY_ACTION)
                .When(SEARCH_WITH, "01068930432",
                      SEARCH_WITH_ACTION)
                .Then(ACCOUNT_INFORMATION_SHOULD_BE_RETURNED, "13120205504",
                      ACCOUNT_INFORMATION_SHOULD_BE_RETURNED_ACTION)

                .Given(AN_ACCOUNT_WITH_PHONE_NUMBER, "01062736745")
                .When(SEARCH_WITH, "01062736745")
                .Then(ACCOUNT_INFORMATION_SHOULD_BE_RETURNED, "12666056628");
        }

        [Scenario]
        public void SearchWithPartialPhoneNumber()
        {
            story.WithScenario("Search with partial phone number")
                .Given(THREE_ACCOUNTS_WITH_PHONE_NUMBER_STARTS_WITH, "0106", EMPTY_ACTION)
                .When(SEARCH_WITH, "0106", SEARCH_WITH_ACTION)
                .Then(ACCOUNT_INFORMATION_SHOULD_BE_RETURNED, "13120205504",
                      ACCOUNT_INFORMATION_SHOULD_BE_RETURNED_ACTION)
                .And(ACCOUNT_INFORMATION_SHOULD_BE_RETURNED, "12666056628")
                .And(ACCOUNT_INFORMATION_SHOULD_BE_RETURNED, "17948552843");
        }

        [Scenario]
        public void SearchWithAPhoneNumberWhichHasSeveralExactMatches() {...}

        [Scenario]
        public void SearchWithNonExistentPhoneNumbers() {...}

        [Scenario]
        public void SearchWithInvalidPhoneNumberValues() {...}

        ...
        ...
    }
    

These test cases are written in C#, but but closer to plain English, so that they are business-readable (please refer to Martin Fowler's BusinessReadableDSL ). With this format, other roles, which have deeper understanding of business domain, can spot missing scenarios or cases.

It's even better if a framework which supports plain text test cases is picked up, like Cucumber in Ruby.

Variables with 'ACTION' suffix are lambda expressions. They give this code life.

SEARCH_WITH_ACTION is for sending the request to the web service and parsing the piped data returned. The code for CustomerService and Subscriber is in the domain layer, because this code is common supporting code for a variety of test cases.

        SEARCH_WITH_ACTION =
            phoneNumber =>
                {
                    subscribers = customerService.SearchWithTelephoneNumber(phoneNumber);
                };
    

ACCOUNT_INFORMATION_SHOULD_BE_RETURNED_ACTION is for verifying the data

        ACCOUNT_INFORMATION_SHOULD_BE_RETURNED_ACTION =
            accountNumber =>
                {
                    //Get expected subscriber from fixture
                    Subscriber expected = SubscriberFixture.Get(accountNumber);
                    CustomAssert.Contains(expected, subscribers);
                };
    

Domain Layer

Class CustomerService is named after the real name of this web service. In the requirements document, daily interpersonal conversation, architecture map, and code, this web service is referred to by the same name. With this unified domain term, the ambiguity is eliminated. For a more complete introduction, please refer to the post Domain Based Testing

    public class CustomerService
    {
        public Subscriber SearchWithTelephoneNumber(string telephoneNumber)
        {
            string url =
                string.Format(
                    "{0}/subscribers?telephoneNumber={1}",
                    endpoint, telephoneNumber);

            //Send http request to web service, parse the xml returned,
            //populate the subscriber object and etc.
            return GetResponse(url);
        }
        ...
    }
    

Class Subscriber models the subscriber. Compared to the piped string format, this tangible format is easier to understand (unless you prefer to refer to telephone number as pipedData[101]?).

    public class Subscriber
    {
        public string AccountNumber { get; set; }
        public string FirstName { get; set; }
        public string Surname { get; set; }
        public string TelephoneNumber { get; set; }
        ...
    }
    

With this domain model, data verification can be carried out based on the object. For example, you can verify that the first name is 'Bei'

    Assert.AreEqual("Bei", subscriber.FirstName);

Or the phone number starts with '010'

    Assert.IsTrue(subscriber.TelephoneNumber.StartsWith("010"));

See the attached test automation source code for a complete working example illustrating the layered architecture. You can open it with Visual Studio 2008, or run it from the command line. You can execute ‘go.bat’ to run the example test, and test results are in the ‘artifacts’ directory. The solution source code consists of three projects. The project with ‘Client’ suffix contains the Domain Layer. The project with ‘Client.Spec’ suffix (spec here is short for specification) contains tests driving the development of this layer (with TDD). The project with ‘Stories’ contains the Test Cases Layer. This source code is tailored from a real project, so you can see some directories containing only one file which is overkill, but necessary if there are more files. Also some classes are returning hard-coded values, just to disconnect from the real system.

How does this solves the problem?

1. Problem: 'Test logic is hard to understand and modify'. Since we have a separated layer focusing only on test logic and making use of supporting code from the layer below, test cases can be expressed in a way that is similar to English, thus, difficulties in reading, reasoning, and modifying test code depend more on the coder's English skill than the code itself.
2. Problem: 'Tests become fragile'. Since we now have a domain layer isolating the test cases from the real system under test, any changes in the system, can be propagated solely to this newly added layer. If we change the code in this layer accordingly, tests cases depending on this layer can still run.
3. Problem: 'Maintenance cost is high'. Thanks to the encapsulation in the domain layer, duplicated code is removed from test cases and you only have to modify one piece of code. Also, since the services and domain models are modelling the system under test, the code is easier to understand and modify.

Frequently Asked Questions

Q: This solution seems complex, do I have to use this?

A: It depends on the size and complexity of the system under test. If the system is very small, and business logic is simple enough, this way is overkill. In this situation, even test automation is a waste of time. If it only takes a couple of minutes to manually test the system, why bother automating tests? For moderately complex systems, mixing test and supporting code can work. If the business logic is complex, I prefer layered architecture.

Q: This architecture requires an investment before real tests can be started, is that wasteful?

A: This is just another way of organizing code. Even if test code isn't organized in this way, code to perform url concatenation, xml/html response parsing, and result verification must be written anyway. With this architecture, you just have to break the code into different classes/methods. Plus, you don't actually have to fully implement these layers all at once. These layers are scenario-driven and test case-driven, and can be implemented as needed.

Q: Designing this requires substantial object-oriented experience, not all QA can do this.

A: I would say test automation is not only QA's responsibility. Other team members, including developers, should contribute.

Developers could focus on the Domain Layer, writing supporting code, making a project-specific platform for QA. And QA, whose talent is designing test cases, write code in Test Cases Layer only. For details, please refer to the post Test Automation - Shared responsibility between QA and Developers.

About the Author

Li Bei (and Bei is his given name) is working in ThoughtWorks as a consultant. He is mostly interested in Domain Driven Design, Test Automation and Domain Specific Language. Bei would like to thank his coworkers. Working with them is always thought-provoking and enjoyable.