This article provides a brief introduction to FlexMonkey and then walks through debugging issues that can be encountered when testing with FlexMonkey.
What is FlexMonkey?
FlexMonkey is an open source tool from Gorilla Logic for testing Flex and AIR applications. The project includes an AIR based console that allows users to quickly create and run user interface tests by providing the ability to record, playback, and verify on Flex visual components. FlexMonkey also allows users to generate ActionScript versions of the tests for easy integration with continuous integration environments.
You can find the FlexMonkey project at: http://www.gorillalogic.com/flexmonkey
Goals / Motivations
FlexMonkey and the underlying automation framework generally work well, especially for the stock Flex components. However, at times, you may bump into problems with record and playback due to custom component extensions, bugs in FlexMonkey, or bugs in the Flex automation framework. This article aims to help you understand how to quickly work through these types of issues.
In showing you how to address FlexMonkey issues, we are going to walk through a typical debugging session, where we automate the testing of a custom component. For the exercise, we will use Justin Shacklette’s Flex 4 Terrific Tab Bar Component. Justin is an engineer at Gorilla Logic who has worked extensively with Flex 4 and has many useful custom components on his blog. The Terrific Tab Bar component adds a close button to the standard Spark TabBar component.
As you will see in this article, it does take some work to deal with the custom TabBar, but it is not as daunting as it may seem on a first look. As with any technology, the better that you understand how it works, the easier it is to solve problems when they come up.
While it can takesome effort to deal with the automation framework, it is well worth the investment, as FlexMonkey allows you to create comprehensive “developer tests” that would be difficult to create by only using unit testing tools. At Gorilla Logic, we’ve moved away from describing user interface testing tasks at a tools level, preferring to avoid terms like “unit testing” and “functional testing”, instead choosing to identify our testing activity by the role completing the work, using descriptions like “developer testing” and QA testing.” By breaking it down this way we can focus on making sure we achieve our different testing goals, but not limiting our developers to a pure unit testing paradigm, when there is such and obvious mismatch between pure unit testing and user interface development.
Even though we believe that developer testing of user interfaces is fundamentally different than other parts of the system, we don’t think that FlexMonkey replaces traditional unit testing tools, like FlexUnit. There are parts of the code base that lend well to traditional unit tests (e.g. business logic, etc). Using both tools provides a killer combination for developers to create a robust developer test suite.
Setting Up
If you want to get follow along at home, you can get started by downloading and importing the Terrific Tab Bar archive file into Flash Builder. Next, setup the project to work with FlexMonkey. The FlexMonkey console will walk you through this setup in the “Project > Properties > Setup Guide” window of the AIR console, as seen in Figure 1. Basically there are two steps: 1) Add the automation_monkey4.x.swc to the Flash Builder project, 2) Update the compile arguments to include the FlexMonkey and Automation libraries. Once you complete these steps, you should be able to launch the Terrific Tab Bar application and the FlexMonkey console. You should see the green connection light on – meaning FlexMonkey and the application are communicating, so it is ready to go.
Figure 1 : FlexMonkey Setup Guide
Debugging Session
In order to do automated testing on the Terrific Tab Bar component some kinks need to be worked out. Here we walk through a chronology of the attempts and fixes to use FlexMonkey to automate the custom component. So, follow along and hopefully when we are done you will understand how to fix your own FlexMonkey issues by using the debugging methods you see here.
If you are new to FlexMonkey, you may want to first read our recent post that gives an Overview on How FlexMonkey Works for background.
Try #1
First, it’s time to do some recording against the Terrific Tab Bar application. I begin by attempting a click on one of the tab labels and see the FlexMonkey console successfully recorded a “select” event. This shows us that FlexMonkey is setup correctly and able to record on standard parts of the Spark TabBar. Next, I click on the close button with less success, as the error in Figure 2 is thrown:
TypeError: Error #1009: Cannot access a property or method of a null object reference. At mx.automation::AutomationManager/isObjectChildOfSystemManagerProxy()[C:\work\flex\dmv_automation\projects\automation_agent\src\mx\automation\AutomationManager.as:2342] at mx.automation::AutomationManager/recordAutomatableEvent()[C:\work\flex\dmv_automation\projects\automation_agent\src\mx\automation\AutomationManager.as:2316] at mx.automation.delegates.core::UIComponentAutomationImpl/recordAutomatableEvent()[E:\SVN\4.x\frameworks\projects\automation\src\mx\automation\delegates\core\UIComponentAutomationImpl.as:387] at spark.automation.delegates.components.supportClasses::SparkButtonBaseAutomationImpl/clickHandler()[E:\dev\4.x\frameworks\projects\automation_spark\src\spark\automation\delegates\components\supportClasses\SparkButtonBaseAutomationImpl.as:129]
Figure 2 : Type Error From AutomationManager
Fix #1: Implement Automation Children Methods
One of the hardest parts of dealing with errors like this from the automation framework is that Adobe does not publish the source code for the AutomationManager class. So, there is not an easy way to see the root cause of this recording issue. When “automation land” seems confused, I typically begin by validating that FlexMonkey and the automation framework are seeing the component tree correctly.
In the latest release of FlexMonkey, it is easy to view the component tree in the FlexMonkey console. With the application connected, all you have to do is open the “Project > View Application Tree” window. This window shows the component tree for the application under test according to “automation land.” As you can see in Figure 3, the close buttons on the tabs do not appear in the tree.
Figure 3 : Application Tree Before Changes
Flex provides three methods for telling the automation framework about the component tree: getAutomationChildern(), getAutomationChildAt(index:int), and numAutomationChildren(). Implementations of these methods are provided on the Flex components, but in some cases it does not see the tree properly. On the TerrificTabBar component, overriding them with the code in Figure 4 causes Flex to see the component’s tree correctly.
override public function getAutomationChildren():Array {
return [_tabButton.closeButton];
}
override public function getAutomationChildAt(index:int):IAutomationObject {
return getAutomationChildren()[index];
}
override public function get numAutomationChildren():int {
return 1;
}
Figure 4 : Implementation of Automation Children Methods
Note: There are two options for where the automation children code can live. It can be added directly to the TerrificTabBarButton class, as it is a UIComponent. Alternatively, it can be implemented on an automation delegate class that we configure to work with the TerrificTabBarButton class. This code logically fits best on the delegate, but sometimes it is easiest to just add it directly to the UIComponent class when it is the only change that is in necessary. The delegate class is the obvious choice for this logic to live when you cannot or do not want to modify the component class source code.
It is time to implement a custom delegate for the TerrificTabBarButton. To do so, I create a TerrificTabBarButtonDelegate class that extends SparkToggleButtonAutomationImpl in a “delegates” package with the contents in Figure 5.
package delegates {
import components.TerrificTabBarButton;
import flash.display.DisplayObject;
import flash.events.MouseEvent;
import mx.automation.Automation;
import mx.automation.IAutomationObject;
import spark.automation.delegates.components.SparkToggleButtonAutomationImpl;
import spark.components.Button;
[Mixin]
public class TerrificTabBarButtonDelegate extends SparkToggleButtonAutomationImpl {
public static function init(root:DisplayObject):void {
Automation.registerDelegateClass(TerrificTabBarButton, TerrificTabBarButtonDelegate);
}
private var _tabButton:TerrificTabBarButton;
public function TerrificTabBarButtonDelegate(obj:TerrificTabBarButton) {
super(obj);
_tabButton = obj;
}
//
// implement automation methods so that Flex knows about about the close button
//
override public function getAutomationChildren():Array {
return [_tabButton.closeButton];
}
override public function getAutomationChildAt(index:int):IAutomationObject {
return getAutomationChildren()[index];
}
override public function get numAutomationChildren():int {
return 1;
}
}
}
Figure 5 : Custom Delegate Implementation
To get this code to compile into the application, I need to tell the Flex compiler about the class because it is not referenced anywhere in the code. This is done by adding the following flag to the compiler arguments in the “Project > Properties > Flex Compiler” screen of Flash Builder:
-includes delegates.TerrificTabBarButtonDelegate
The three automation children methods tell the automation framework that the TerrificTabBarButton actually does have a child, unlike the standard TabBarButton. The standard TabBar button does not have a child, and thus returns a null Array for the getAutomationChildren() method. With this change, the Flex automation framework should be able to see the close button as a child of the custom TabBar button.
Try #2
Now, that the code is updated, I reload the application and the “Project > View Application Tree” window of FlexMonkey with much better results. Figure 6 shows that FlexMonkey is now showing the component tree correctly, as there is a close button for each tab.
Figure 6 : Application Component Tree After Updating Automation Children Methods
Next, I attempt to record on the close button again. Unfortunately, the error from Figure 1 is still thrown from inside the AutomationManager class.
Even though overriding the automation children methods did not do anything to solve the problem with the custom TabBar, it still is an important part of the automation / FlexMonkey puzzle to understand. Many automation problems are resolved by working with these methods. It is also worth noting that it is not always about telling the automation manager about components it cannot find. At times it can be just as fruitful to hide components from the automation manager by overriding these methods. It is really just something you have to deal with on a case-by-case basis. With the new FlexMonkey “Application Automation Tree” interface, it is easy to see what the automation framework sees. In most cases it should be apparent what direction you want to try taking things.
Fix #2: Hide Event From Automation Manager
At this point, I take a closer look at the stack trace. The source code for the AutomationManager class is unavailable, but Adobe does provide source code for the delegate classes. So, debugging on their source code can give a better idea of why recording is failing.
To add the source code, click on the “Edit Source Lookup Path…” button that is displayed in the file window that opens when clicking on the delegate class in the stack trace. The delegate source code can be found in “[SDK HOME]/frameworks/projects” directory. There are multiple projects in that directory. For the delegates in this stack trace, I added the following directories: “automation/src” and “automation_spark.”
In hopes of understanding what is happening, I debug into the “clickHandler” method of the SparkButtonBaseAutomationImpl and inspect the event as seen in Figure 7. The event looks as expected, but I do note that it has a “target” of “spark.components.Button,” which is what I expect as the close button is of that type.
Figure 7 : Debugging Into Delegate Class
If I debug into this method again, but click on the label portion of the tab instead of the close button, the “target” for the event is a Label instead of a Button. Rather than fighting with “automation land” when I do not have the information to do so (i.e. the AutomationManager source code), I settle on suppressing the event causing the error from the making it into “automation land.” I do this by overriding the “clickHandler” method in the TerrificTabBarButtonDelegate class we created in Fix #1 by adding the code in Figure 8 to the delegate class.
override protected function clickHandler( event:MouseEvent):void {
if(event.target.automationName != "closeButton")
super.clickHandler(event);
}
}
Figure 8 : Override clickHandler to Suppress Event
The “clickHandler” override checks the event target and only calls the super classes implementation if it is not for the close button.
Try #3
I refresh and try recording on the close button again. Finally, some real success! The error is no longer thrown. However, FlexMonkey is recording a “Select” event when we click on the close button, which if we playback will only select the tab, not close it as should happen.
Fix #3: Suppress Extra Select Event in the TabBar Component
At this point, while things are not yet working, the code is in pretty good shape because the mysterious error has been removed from the mix. Now that it is possible to plot a course, I plan to create another delegate class. This delegate will be for the custom TabBar. Similar to the TerrificTabBarButtonDelegate implementation, I am going to suppress the event that is causing the “Select” action to be recorded when clicking on the close button. Then, in Fix #4, it is time to actually automate the custom TerrificTabBarEvent for recording and playback, which will be the final piece to getting the component setup for working with FlexMonkey.
Repeating the same steps followed in Fix #1, I create another custom delegate class called TerrificTabBarDelegate. The code for the new delegate can be seen in Figure 9. As you can see, this code is very similar to the first delegate for suppressing the event.
override protected function clickHandler( event:MouseEvent):void {
if(event.target.automationName != "closeButton")
super.clickHandler(event);
}
}
Figure 9 : TabBar Custom Delegate
If you are following along at home, do not forget to add the compile argument to include in the new delegate class, as the Flex compiler only includes classes that are referenced elsewhere in the code:
-includes delegates.TerrificTabBarDelegate
Try #4
As I hoped, now when recording a click on the close button, nothing happens. So, we now have delegate classes in place and nothing unintended is happening during recording. I can now move on to the easy part, automating the custom event that will actually allow me to record and playback on the custom TabBar.
Fix #4: Instrument Custom Close Event
It is not always “normal” to have to deal with suppressing events as I have done in this article. However, it is a fairly normal activity to have to instrument custom components to work properly with FlexMonkey, and it is actually much easier than it sounds. Basically, I am going to add an event handler for the custom event in the delegate, and then tell the delegate to record and playback whenever it happens. The only other piece of work is that the custom event has to be described in the FlexMonkeyEnv.xml environment file.
To begin, I am going to create a custom version of the TerrificTabBarEvent to use for record and playback. I need to do this because the TerrificTabBarEvent class has a constructor with custom required arguments. An alternative to the custom event I am creating next would be to give the “index” property a default value in the constructor and make it writeable with a setter. However, I want to show here how this can be automated without modifying the code being tested.
Using the contents of Figure 10 I create the custom event class AutomationTerrificTabBarEvent in the “delegates.events” package. This event allows FlexMonkey to instantiate an event with just the “type” value and then set the “index” as property outside of the constructor. The “index” property is a custom event property that tells the TerrificTabBar component which tab to close.
package delegates.events {
import flash.events.Event;
public class AutomationTerrificTabBarEvent extends Event {
public var index:int;
public function AutomationTerrificTabBarEvent(type:String, index:int=-1, bubbles:Boolean=false, cancelable:Boolean=false) {
super(type, bubbles, cancelable);
this.index = index;
}
override public function clone():Event {
return new AutomationTerrificTabBarEvent(type,index,bubbles,cancelable);
}
}
}
Figure 10 : Update Event Class
Next, we are going to modify the FlexMonkey environment file to tell it about this event. Another utility window that was added in the latest release of FlexMonkey is the “Project > View Environment File” option, which shows the current file contents in use as seen in Figure 11. This can be useful for confirming that any custom edits are being picked up and for accessing the current file contents for editing. To create my own custom file, I copy the contents from that screen and paste them into a file called FlexMonkeyEnv.xml in the same directory as my MXML application source file. This causes Flash Builder to copy it out to the same “bin” directory as my SWF file, where FlexMonkey can load it and use it instead of the default version of the file.
Figure 11: FlexMonkey Environment File Window
Once the file is created, I add the snippet from Figure 12. This is needed to tell the automation framework about the custom AutomationTerrificTabBarEvent and its properties.
<!-- Terrific Tab Bar --> <ClassInfo Name="TerrificTabBar" Extends="SparkListBase"> <Implementation Class="components::TerrificTabBar"/> <Events> <Event Name="CloseTab"> <Implementation Class="delegates.events.AutomationTerrificTabBarEvent" Type="closeTab"/> <Property Name="index"> <PropertyType Type="int"/> </Property> </Event> </Events> </ClassInfo>
Figure 12: Environment File Updates
Looking at the environment file can be a bit misleading, as it initially appears that you need an entry for any custom delegates you add. But, as you may have noted, I have not had to edit this file previously. This is because this file describes a hierarchy of the components, so an entry is only necessary for things that cannot be found on the parent class descriptions. The best example for understanding this is to look at the entry for the FlexDisplayObject, which describes the “Click” event. Since all Flex components extend UIComponent (the class for this entry), the automation library has a description for click events that don’t need repeated on each entry.
The next step is to add logic to the delegate class for recording and playing back on the event. In the constructor, we add event listener to DisplayObject that is passed in (an instance of TerrificTabBaButton). In code for Figure 13, I update the delegate constructor to add an event listener for the “closeTab” event and keep a private reference to the TabBar component. The event handler priority is set to default plus one so that “automation land” sees the event before standard handlers.
private var _tabBar:TerrificTabBar;
public function TerrificTabBarDelegate(obj:TerrificTabBar) {
super(obj);
_tabBar = obj;
obj.addEventListener(TerrificTabBarEvent.CLOSE_TAB, closeHandler, false, EventPriority.DEFAULT+1, true);
}
Figure 13: Add Delegate Event Handler
Next, I implement the handler as seen in Figure 14. This tells FlexMonkey to record the event using the custom AutomationTerrificTabBarEvent I created previously.
protected function closeHandler( event:TerrificTabBarEvent):void {
recordAutomatableEvent(new AutomationTerrificTabBarEvent(TerrificTabBarEvent.CLOSE_TAB, event.index));
}
Figure 14: Delegate Event Handler
After setting up the delegate for recording, the next step is to add the code in Figure 15 which tells the delegate how to playback the event, which dispatches the TerrificTabBarEvent on the proper component to get the desired playback results.
override public function replayAutomatableEvent(event:Event):Boolean {
if(event is AutomationTerrificTabBarEvent) {
return _tabBar.dataGroup.dispatchEvent(new TerrificTabBarEvent(TerrificTabBarEvent.CLOSE_TAB, (event as AutomationTerrificTabBarEvent).index));
} else {
return super.replayAutomatableEvent(event);
}
}
Figure 15: Delegate Playback Logic
Try #5
At last, I can now successfully record and playback on the close clicks custom TabBar!
Solutions
With the record and playback finally working, it’s time to review the solutions implemented. In the end, I have only a handful of changes with two delegate classes, a custom event class, and some minor modifications to the FlexMonkey environment file. Here is the recap of what I did:
- Overrode the automation children methods so that FlexMonkey and the automation framework could see the close button on each of the tabs.
- Prevented the close button click event being dispatched on ButtonBarButton from FlexMonkey / automation framework because it was causing an error in the AutomationManager class.
- Prevented the close button click event being dispatched on TabBar from FlexMonkey / automation framework because it was being recorded as a “Select” event, which was not appropriate for this case.
- Automated the custom TerrificTabBarEvent in the TerrificTabBarEventDelegate so that FlexMonkey would be able to record and playback on the component.
Resources
- An archive of the final Flash Builder project from the “Debugging Session” section can found at: http://gorillajawn.com/flexmonkey/TerrificTabBar.zip
- You can find out more about FlexMonkey and Gorilla Logic, Inc at: http://www.gorillalogic.com/
- The FlexMonkey project blog can be found at: http://flexmonkey.org .
- More information on the automation framework can be found at: http://help.adobe.com/en_US/.../-7ec5.html
About the Author
Jon Rose is an enterprise software consultant and Flex Practice Director at Gorilla Logic, Inc. (www.gorillalogic.com) located in Boulder, Colorado. He is an editor and contributor to InfoQ.com, an enterprise software community. He is the co-host of DrunkOnSoftware.com, a videocast for those who like booze and bits. He has worked with clients large and small in both the private sector and government. His love of solving problems drives him to build quality software. You can read his blog at: http://gorillajawn.com.