UI Testing in F# with canopy

Although Selenium is a popular library for UI testing, issues about fragile and unreliable tests are common. InfoQ reached out Chris Holt, creator of canopy, to learn more about the F# library built on top of Selenium.

InfoQ: What is canopy?

Chris Holt: Canopy is an F# layer on top of Selenium with the goal of making UI testing behave the way you expect it to. Selenium works really well but can be too literal at times. With canopy, all of the actions aren’t click an element and fail if you can’t, they are instead try really hard to click and fail if you can’t after a reasonable amount of time. This helps you create less brittle tests that work the first time more often.

InfoQ: How does it make UI testing with Selenium easier?

CH: Built in retry-ability, both in retrieving elements, but also in the actions you take on the screens as well as validation. There are also helpful error messages that guide to correct solutions for common problems like misspelling selectors. Various ways of grabbing elements are supported seamlessly and it’s easy to extend this capability.

For example if you have a button on your screen with the text Save you could simply do click "Save" and it would work. With regular selenium you have choose ByText, ById, ByCSS, ByXPath, etc. You can extend this by adding finders for common idioms in your webpage like placeholder values in form data, or arbitrary data-* tags that have valuable meta data.

Canopy also has a terse API so you can both read an write tests more easily. It also smoothes over differences in html. For example, each input type have different representations in html, and the way you manipulate them in raw Selenium is different. They are all the same in canopy. For example:

// Assign a value to a textbox or dropdown
"#state" << "New York"

InfoQ: Does canopy work with external automation services such as Browserstack?

CH: Yes, anything that Selenium works with, canopy works with too. For Browserstack you would use a RemoteWebdriver. Since canopy has a lot of built in retry functionality it is more chatty. This will usually be acceptable though as you won’t need to use Sleep as often. There are also opt-in optimizations in canopy if you want to be specific about your selector types and not rely on the figure it out for you mode.

InfoQ: Is there a recommended way for selecting elements on a page? For example, does selecting elements by id leads to more maintainable/robust tests?

CH: Most of the ‘skill’ in UI Automation comes from selecting elements. There is a balance between clean and sane selectors and having to add additional attributes like classes or ids to your markup. I find that CSS/JQuery selectors are the best syntactically; you can use them for about 80-90% of scenarios. XPath can be used in the remaining 10-20%. Things like exact text matches, and finding the parent of an element require XPath. By value/inner text (built in) is also very handy like in the click "Save" example above. It uses XPath behind the scenes.

After some practice you will get good at creating selectors, and I suggest learning to do them by hand instead of using a tool to generate them. They will be more accurate and more resistant to changes in page structure affecting your tests. After you get a hang of selectors you will learn how to make your html testable which helps out a lot.

Selectors can often be written in a way that is easy to understand, for example #header .links to get the links in the header div of the page. In auto generated xpath this may read something like html/body/div/div/div[2]/ul/li/a which does not help at all in understanding what it means, and if a single div is added/removed in the chain, the selector will break. With the css one it will work ‘forever’ unless someone changes the Id of header, or removes/changes the links class.

InfoQ: Is it possible to customize the error reporting? For example, how could we take a screenshot when a test fails?

CH: There are currently three built-in reporters for canopy: ConsoleReporter, TeamCityReporter and HtmlReporter. Adding a custom one simply requires to implement the IReporter interface.

InfoQ: What are the different ways to extend canopy?

1) Customize the test output by implementing IReporter.

2) Add some common finders to the set of finders that canopy uses to try to find the element for you.

3) Write new functions for actions you use frequently. Because of F# will run the most recently defined version of a function, you can also ‘overwrite’ existing functions to do what you want.

You could create your own version of ‘click’, for example. Just create a module, something like ‘canopyExtensions’ and ‘open’ it after doing ‘open canopy’ and put all your extensions and overrides in there. Now all your tests will use your custom stuff as you define it without having to change any of them.

Here is a good example of where someone wanted to be able to ctrl+click items in a multi-select check box. Canopy did not have this feature, so they added it to their extensions.

4) Canopy does not hide or abstract any part of selenium. It uses IWebDriver and gets/uses IWebElement. Any articles you find or questions on Stack overflow are 100% applicable and usable with canopy. The only thing needed is to translate the code to F#.

InfoQ: What are the steps required to run canopy tests on a CI server?

CH: The tests run by executing the console application resulting of the build. Differents output formats are supported for the tests restuls. The configuration using TeamCity is literally a one liner:
reporter <- new TeamCityReporter() :> IReporter

I personally use Rake/fake tasks to do all my building, so my TC task would be a simple as a fetch from source control, then a command line statement to kick off my task that does the rest.

For Jenkins I use the HtmlReporter. There is a plugin for Jenkins that can pick up an html document and save it with the job result. That too is just a few lines to setup.

Canopy is an open source project hosted on GitHub. For a longer introduction, Chris Holt recently talked about canopy at fsharpConf.