FlexMonkey brings unit testing to Flex user interface developers

The practice of maintaining automated unit test suites has gained widespread acceptance over the past decade to the point where most developers today either engage in some amount of test writing or at least feel bad for not doing it. This rise in automated unit testing has led to some confusion about who should be testing what. Should developers strive for 100% code coverage with their unit tests, and if so, does that mean we no longer need dedicated QA testers? Many development teams draw the line at the user interface, reasoning that since user interfaces can be exercised with little or no programming they can be tested more economically by dedicated testers, either manually or with specialized testing tools. This division of labor has led many to divide the world of testing into "unit testing" and "functional testing", with developers providing the former and QA testers providing the latter. In this article we'll explore how Gorilla Logic's new, open source Flex user interface automation testing tool, FlexMonkey, can enhance the productivity of both developers and QA testers. FlexMonkey allows developers to incorporate user interface testing into unit test suites and continuous integration environments, and allows QA testers to expand those unit tests to create and maintain comprehensive quality tests.

While certainly the testing conducted by both developers and QA testers is ultimately aimed at making sure an application works as expected, the overarching goals of development testing is somewhat different from QA's goals. The goal of developer testing is to ensure that newly written or modified code works as intended and doesn't cause "regression" (ie, the inadvertent breaking of previously working code). Automated regression testing is a critical part of agile development, since only an insane developer would do any but the most trivial refactoring of working code without having enough automated testing to ensure that the refactored code still worked.

It is reasonable to expect a developer to test every newly written piece of code, but it is not necessary that each such test be automated. Manual testing can be completely effective for testing new or modified functionality, but manual testing is almost always impractical as a means of guarding against regression. To prevent regression, developers must provide a set of tests that exercise an application top-to-bottom and end-to-end (front-end to back-end), but anecdotal evidence suggests that even automated tests covering as little as 50% of the code can effectively guard against regression in many applications.

QA testing is far more exhaustive than developer testing in that it aims to ensure that code works properly across every conceivable usage scenario. To put it another way, QA testers are on a mission to break an app by doing horrible things to it. Developers of course are familiar with doing many horrible things and could provide such testing themselves, but this would leave very little time for actually writing code.

Competent developers either conform to a test-first discipline in which small test harnesses are built to exercise an API before the API itself is even implemented, or else employ a code-a-little-test-a-little (CALTAL) approach in which each unit of code developed is tested immediately after its compiled. The CALTAL approach is especially prevalent among user interface (UI) developers, since unlike an API which can be defined as a logical interface prior to implementation, there is no real means of expressing a user interface for the purposes of testing without actually implementing some amount of view and controller code.

Traditionally, programmers have mainly relied on the xUnit family of testing frameworks to organize and execute unit tests. xUnit frameworks, such as JUnit and FlexUnit, let developers manage large suites of automated tests. Using these frameworks, individual test suites can be run separately or combined so that a suite developed for one aspect of a system can be run standalone, or included as part of a larger suite that tests a complete subsystem or the entire application, and consolidate the reporting so that developers and management can easily review summary and detailed test results. xUnit tests are themselves small programs that exercise application code and verify that actual results match expected ones. xUnit test suites provide a simple but effective means of guarding against regression. Most build systems, such as ant, provide direct support for running xUnit test suites as part of the application build process, and continuous integration systems, such as Cruise Control or Hudson, kick off such builds automatically each time code is committed to the team's version control system, and helpfully identify any person deserving to be ridiculed by the team for committing code that fails the test suite. (Such ridicule is critical to increasing the overall effectiveness of a development team.)

QA testers, on the other hand, have mainly relied on visual tools that provide for the recording of interactions with an application such as mouse clicks and keyboard input, and that can then play back those interactions while verifying results. Historically, such tools have not been well-suited for use by developers since the resulting recorded scripts are often very brittle, such that minor changes to a view's layout like reordering input fields or the introduction of additional user interface elements unrelated to a particular script can cause playback to fail, even though the usage scenario is still logically compatible with the updated view. As a result, UI developers have tended to "peel away" the user interface itself when automating UI testing by writing tests that programatically exercise the controller code by calling it directly, or by generating "synthetic" UI events. While this approach is not without merit, it can be difficult to accurately reproduce the asynchronous processing of user interface code. As a result, it can be labor-intensive to develop such tests, and since the user interface itself is not being directly exercised by the tests, it is often impossible for such tests to verify correct handling of keyboard and mouse input or the correct display of data. It is consequently common for all developer testing of a user interface to be a manual process. While such an approach certainly allows a UI developer to engage in a CALTAL process, it results in the user interface being completely left out of automated regression and continuous integration testing.

FlexMonkey was designed from the ground up to provide robust recording and playback functionality similar to commercial tools such as HP's QuickTest Professional, but to do it in such a way that it can meet the needs of both developers and QA testers. FlexMonkey provides for the interactive recording and playback of user interaction scenarios, but can additionally save tests as ready-to-run FlexUnit test suites. The generated ActionScript comprising these tests is readable and editable, and can even be coded from scratch rather than through recording.

Interactively Creating Tests

FlexMonkey is itself an Adobe AIR application, but can test both AIR and Flex applications, including server-based Flex SWFs that use remote services such as BlazeDS. FlexMonkey can record and playback interactions for an application without requiring the linking in of any special testing libraries, and applications can be tested while running standalone or within a browser.

To create a test suite, you use the FlexMonkey launcher to run the application, either by navigating to a SWF file via a file selection dialog, or by specifying a URL from which to load a SWF residing on a server. FlexMonkey will launch a SWF and display it in its own window or within a browser page, and will open a separate window containing the FlexMonkey Console which manages the recording and editing of user interface tests. To record a test, you simply click on the Record button and then interact with your application. As you click on components and type into fields in your application's window, each action is recorded and displayed in the FlexMonkey Console. The screen shot below shows the results of a short recording session in which we use a simple contact manager application ("Monkey Contact Manager") to enter a name ("Fred"), and phone number ("555 555 1212"), and click the Add button to create a new row in the application's contact list. FlexMonkey can faithfully record any UI Event, including more "exotic" events such as selecting a value in a ComboBox that's embedded in a DataGrid.

In the screen shot above, we see the FlexMonkey Console containing events recorded while manually testing the Monkey Contact Manager application.

FlexMonkey uses the familiar xUnit testing hierarchy which organizes tests into test cases and suites. In this example, we've created a suite called MonkeyContactsSuite containing a single test case called MonkeyContactsCase. The test case, in turn, contains a single test called TestAddNewContact, which is made up of the various user actions we have just finished recording. For each user action, FlexMonkey captures a UI Event, a property and value pair that identifies the target of the event, and event-specific arguments. By clicking on the first Input action, we can see its details displayed on the right side of the FlexMonkey Console, and we can edit the various arguments. We could, for example, change the script to input "Ethel" rather than "Fred" during playback.

For this action, an Input UI Event with an argument of "Ethel" is sent to a component having an automationName property with a value of inName. In other words, this action directs FlexMonkey to enter "Ethel" into the Name field of the contact manager during playback. The Container Value and Container Property fields can be used to further qualify which component is an event's target by specifying a property/value pair that identifies a parent container. This ability to identify UI components by target as well as container properties makes it easy to unambiguously reference any UI Component, even ones for which no automationName property has been specified. We could for example create an action that clicks on the component with label="Go" inside the component with id="editPanel".

We can interactively create an assertion to validate some set of property values by clicking on the Verify button (the green checkmark) on the FlexMonkey Console, which opens the FlexMonkey Spy Window. Using the spy window, we can select any set of properties of a UI Component for which we wish to verify some expected values. The screen shot below shows how the spy window can be used to specify that the Name field's text property should have a value of "Fred" after a set of actions have been executed.

In addition to verifying property values, FlexMonkey can capture bitmap images of selected area of an application window during recording, and compare the same area of a window to this bitmap during playback, providing a way to automate the "eyeballing" of select portions of the application's actual screen presentation.

We can now run our test by clicking the Play button. FlexMonkey replays each recorded action back against the contact manager application, and the application responds exactly as if those actions were being manually entered with the mouse and keyboard. The result of our simple test is shown below.

Using FlexMonkey in this way we can interactively create and maintain various, complex test suites consisting of any number of test cases and tests.

Generating Tests in ActionScript

In addition to interactively creating and running tests, we can use the FlexMonkey API to specify tests programatically that can be run with xUnit framewords including Fluint (http://fluint.googlecode.com) and FlexUnit 4 (http://opensource.adobe.com/wiki/display/ flexunit/FlexUnit).

Although it's quite reasonable to create tests entirely from scratch by coding against the FlexMonkey API, we can use the FlexMonkey Console instead to create ActionScript-based tests directly from our recorded test scenarios. Selecting the "Generate AS3" menu option creates ActionScript classes for each suite and test case. Each individual test is output as a method within its corresponding test case class. Here is the generated test case class for our simple example.

package testSuites.MonkeyContactsSuite.tests {
    import com.gorillalogic.flexmonkey.events.MonkeyCommandRunnerEvent;
    import com.gorillalogic.flexmonkey.core.MonkeyTest;
    import com.gorillalogic.flexmonkey.monkeyCommands.*;
    import com.gorillalogic.flexmonkey.application.VOs.AttributeVO;
    import com.gorillalogic.flexmonkey.flexunit.tests.MonkeyFlexUnitTestCase;

    import mx.collections.ArrayCollection;

    import flash.events.IEventDispatcher;

    public class MonkeyContactsCase extends MonkeyFlexUnitTestCase{
	public function MonkeyContactsCase(){
	    super();
	}

	private var mtTestAddNewContact:MonkeyTest = new MonkeyTest(null,
'TestAddNewContact', null, 500,
	    new ArrayCollection([
		new UIEventMonkeyCommand('Input', 'inName', 'automationName', ['Fred']),
		new UIEventMonkeyCommand('Input', 'inPhone', 'automationName', ['555 555 1212']),
		new UIEventMonkeyCommand('Click', 'Add', 'automationName', ['0'])
		]));

	private var mtTestAddNewContactTimeout:int = 8500;

	[Test]
	public function TestAddNewContact():void{
	    // startTest(<MonkeyTest>, <Complete method>, <Async timeout value>,
<Timeout method>)
	     startTest(mtTestAddNewContact, TestAddNewContactComplete,
mtTestIA1TimeoutTime, defaultTimeoutHandler);
	}

	public function TestAddNewContactComplete(event:MonkeyCommandRunnerEvent, passThroughData:Object):void{
	     checkCommandResults(mtTestAddNewContact);
	}
    }
}

A UIEventMonkeyCommand has been generated for each action in our test. Each of these commands specifies an action, a property/value pair identifying the target component for the action, and an array of action-specific arguments. (All available actions are documented at http://code.google.com/p/flexmonkey/wiki/FlexMonkeyCommands.) In the example, each UIComponent has been identified with an automationName property but, as mentioned above, we could instead use any property/value pair.

We can easily modify the script without re-recording. For example, we could select a Phone Type of "Mobile" from the contact manager's ComboBox by inserting the following command into the array:

new UIEventMonkeyCommand('Select', 'inType', 'id', ['Mobile'])

We could test adding 1000 contacts to our manager by generating commands programatically. For example, the lines of code below repeatedly enters a unique contact name and hits the add button:

for (var i:int=0; i<1000; i++)
    { mtTestAddNewContact.addItem(new UIEventMonkeyCommand('Input', 'inName', 'automationName', ['Contact' + i]));
    mtTestAddNewContact.addItem(new UIEventMonkeyCommand('Click', 'Add', 'automationName', ['0']));
}

For each generated test method, a corresponding Complete method is also generated. The Complete method is called upon completion the test method. We can add additional assertions to this method to test additional properties. For example, we could check to see that when the script is completed the name of the first contact in the list is "Fred".

public function TestAddNewContactComplete(event:MonkeyCommandRunnerEvent, passThroughData:Object):void{
    var grid:DataGrid = MonkeyUtils.findComponentWith("grid", "id"); // Find the component with id="grid"

    var actual:String = grid.dataProvider[0]["name"] assertEquals("First contact was not Fred", "Fred", actual);
}

We can in this way test values of non-visual application classes as well.

We can run our tests using FlexMonkey's GUI TestRunner which loads the SWF to be tested as well as the SWF containing the tests themselves, and then runs the tests.

Although the example we have examined here is extremely simplistic, it should be clear that FlexMonkey provides the convenience and efficiency of user interface scenario recording with the power and flexibility of code generation and customization.

Continuous Integration for UI Testing

FlexMonkey provides a test runner that can be invoked from an ant task and that outputs results that can be rolled up with other FlexUnit tests as well as JUnit tests for Java. Using JUnit's junitreport ant task, FlexMonkey test output can be rendered in the familiar JUnit report format shown below.

The FlexMonkey ant task starts the FlexMonkey launcher which loads the SWF to be tested and the SWF containing the tests themselves. The output is sent over a socket back to the FlexMonkey ant task which writes them to disk so that they can be merged with other Fluint/FlexUnit or JUnit test output to provide consolidated reporting of all Flex and Java tests comprising a suite.

Because FlexMonkey can be run from ant, it can be easily integrated into continuous integration builds using frameworks such as Cruise Control or Hudson.

Top-Down Testing with the Monkey

Through combining a recorder, a code generator, and an easy-to-use API, FlexMonkey allows developers to create sophisticated, automated user interface tests tailored to exercise application functionality without being brittle with respect to the ongoing evolution of a user interface throughout the development lifecycle. In Flex applications, the user interface is the top of the stack for most operations, and FlexMonkey allows for a top-down approach to testing that provides a powerful complement to unit testing of lower level application classes. By creating tests that exercise the user interface itself, developers can test the maximum amount of code with the fewest number of tests. In fact, by running FlexMonkey tests against an application that is communicating with a remote back-end, FlexMonkey can drive end-to-end testing from the user interface all the way back to databases or external systems. Including such tests in a continuous integration environment provides maximum assurance for the entire team that each code change has not inadvertently broken some distant, but dependent, part of an application system.

Developer Testing vs QA Testing

While a developer could certainly use FlexMonkey to create tests that provide complete code coverage and entirely obviate the need for QA testing, it continues to be a better division of labor and skill specialization to have developers automate representative application functionality, but leave it to QA to provide the bulk of the testing required to ensure complete code coverage, as well as to more generally do horrible things to an application to see how it stands up. FlexMonkey meets the needs of both developers and QA testers, and by providing a common testing platform to both allows tests initially created by developers to provide a basis for more extensive tests created by QA testers.

It is additionally possible to integrate QA tests into a continuous integration environment, although the blanket nature of QA tests can sometimes make them too brittle to be included as part of a continuous integration suite. In any case, having the option to add selected QA tests to an integration suite can make it much easier for developers to establish a comprehensive base of tests.

Getting Started

You can find links to FlexMonkey source, binaries, and complete documentation at http://flexmonkey.gorillalogic.com. Gorilla Logic also offers training and consulting services that can jumpstart user interface testing automation, as well as help to quickly establish comprehensive continuous integration environments. You can also follow the project blog: http://flexmonkey.org.

About the Author

Stu Stern is President & CEO of Gorilla Logic, an application development firm specializing in Flex and Java. Prior to founding Gorilla Logic in 2002, Stu was Vice President of global Java consulting services at Sun Microsystems. Before joining Sun, Stu was Vice President of Equity Trading Systems at PaineWebber (now UBS).Although he crossed over to the management dark side fairly early in his career, Stu has managed to keep writing code. You can read Stu's blog at http://stu-stern.blogspot.com.