The Key to the Perfect Mobile Application
You just spent the better part of last quarter developing the latest version of your mobile application. It was a must. The recent upgrades in the major mobile operating systems enabled new capabilities to your mobile applications. Your competitors have already capitalized by extending the functionality of their applications. Their promotional blitz about what their app can now do has begun. With every commercial your application looks more and more like a relic from the Stone Age.
Your job is to make sure your company's app doesn't fall behind. The pressure to release your latest version as soon as possible is the biggest risk in mobile application development. It's the reason why the majority of apps are one starred, or deleted after only a handful of uses. They're broken.
Mobile application testing is the process of catching all of the functional and performance issues of your mobile app before your customers do. Even if you beat your competitor to market, if there are problems discovered by the end user you will have wished you waited. All it takes is an angry user with a large social following and thousands of man hours along with millions of dollars have been wasted.
This is what makes mobile application testing the most critical part of the application development process. It's a lot like a beating heart. You don't feel it too often, but if it isn't working right the consequences can be catastrophic.
Not Your Father's Application
A desktop application had a simpler matrix to test. The location of a desktop computer was fixed, the screen size was standard, and the operating system was usually the same. Mobility has changed all the rules.
"The internet gave a company a chance to say, 'We're always open for business.' Now the mobile web let's a company say, 'we're open anywhere, anytime.' There is a lot more work involved to support the bigger claim." says Tal Barmeir, CEO of Experitest.
The test matrix for mobile is more complex. A smartphone, and all of the applications it carries, goes from place to place, changing local networks all the time. A businessman travelling from Beijing to Barcelona takes his application across continents. An application that works smoother in some places than others can irritate its users. To even out performance over different locations, the app must be tested through different levels of bandwidth, latency, jitter, and other variables that change with each local network.
From wearables to tablets mobile devices come in varying shapes and sizes. The screen can be all sorts of dimensions. This brings new challenges. What looks great on a desktop may not look so good on a tablet, and even worse on a smartphone. Even functions must be tested across different sizes. What if a button is too small for a user’s fingers? What if important information is not as visible when compressed into a mobile screen? These issues are compounded by the vast number of mobile devices available to consumer. According to a report published in August 2014 by OpenSignal, there are almost 20,000 types of devices with their own dimensions and resolution in the Android brand alone.
There are also four main operating systems, all with unique configurations. A standard test set for mobile application testing can be a multidimensional matrix consisting of millions of combinations of locations, device types, sizes, and operating systems.
Nobody Has All Day
This complicated matrix is why mobile testing can be the slowest moving phase of the development process. A testing team has to make sure every function of their mobile application is working properly and smoothly, without pauses, crashes, or too much buffering – in every location on earth. A job like this takes time to complete.
Unfortunately, the nature of mobile application development doesn't allow a testing team to take all the time it needs. Competition is so fierce that every day a competitor has the latest version of its application on the market that includes features not part of your app, that application runs the risk of becoming obsolete. Every day the technological advantage sticks is one your app is fades towards oblivion.
This puts a DevOps team in a difficult spot. They can test thoroughly, but they risk releasing their mobile application behind the competition. There is a temptation to do something else, cut corners. A manager can make a decision to cut short the testing process. He can decide to test a fraction of what he should test to bring their app to market quicker.
It's a game of Russian roulette. The risk is that instead of the quality assurance team catching the imperfections in the application, it is the customers themselves that make up the "auxiliary QA team”. Anything not tested or fixed can give the end user reason to delete the app and never do business with the company again.
What to Test for
The two main components of an application are functionality and performance. Does every function of the application work? Are there any scenarios where when you use the application one way it crashes, gives you an error message, or doesn't do what you expected? Performance is speed, and speed is increasingly becoming the new functionality. If any area of a mobile application performs too slowly, in the eyes of the end user it's broken. This can get challenging, because if an application isn't working right, or sluggish due to conditions that the developer does not control, it's still their fault. The user will assume the problem lies with the mobile application and considering deleting it. It may not be fair but it's the rules of the trade.
A tester has to test for everything they can control, and as much of what they cannot control as possible. They need to simulate the same conditions for any local network on earth, modulating bandwidth, latency, and reception. The tester must make sure his application performs under these conditions effectively. Along with testing locations, they need to test positions. What happens when the user enters an elevator, or drives through a tunnel?
The Evolving Testing Environment
Mobile testing has evolved very quickly, all of it aimed at reducing the time it takes to test. The first type of testing was manual testing. A user would download their application onto a device and physically use it. Then the tester would test each function by executing it like a user, using their hands to type text, and press buttons, a very slow process. The tester would be limited to testing one device at a time. Manual testing has improved to the point where you can plug in a device and manipulate it with your mouse and keyboard.
The next step is test automation. Automation enables you to test on multiple devices all at once. A tester chooses one device to create a test script. Using the keyboard and mouse, you begin to perform a series of steps on the application using a reflection of the screen on your desktop. Once you are done recording, the testing software converts the sequence of steps into a test script. The script can then be automatically run on all of the devices, generating reports on the results for each test. The best testing tools are able to generate scripts in all the major test scripting languages, converting them from one to another instantly.
Once the scripts are made, they can be edited to create loops, if statements, or any programming additions to enhance the value of the test. The script integrates into any Continuous Integration (CI) server, which will merge the testing steps into the development process with minimal amount of manual work.
Beat the Clock with Cloud Testing
The greatest leap in expanding test coverage and reducing testing time has been the introduction of cloud testing. Setting up a mobile testing tool inside the cloud has reaped tremendous productivity gains in both manual and automated testing, as well as furthering the impact of an agile development process.
Here's how it works: A cloud testing tool can be attached to a server. A series of machines, all with devices connected to them, connects to the server. Through these machines you can connect as many devices as you want. It creates an on premise mobile device lab where a tester can access any device connected. It's that simple.
This allows for every mobile application tester to access all of the devices a business decides to connect. It can be 20 devices, or it can be 200 devices, or more. Each device can have its own size, version, and operating system, creating a matrix of combinations large enough to cover a substantial test area.
Cloud testing is ideal for agile development. A tester can be assigned to a specific project at a remote location along with a handful of members from each part of the DevOps department. Accessing the cloud, the tester can have at his fingertips all of the devices available in the company. Even as he moves from one company office to another he takes the entire mobile testing apparatus with him.
Cloud testing expands the scale of manual and automated testing. Instead of being limited to the devices a tester can connect to his personal machine, they can use his manual or automated testing tool on any device in the cloud by remotely connecting to it, and manipulating the device through a reflection of the device's screen on his desktop's screen. If he needs to test for a specific device, using a specific version of a specific operating system, he doesn't need to worry about finding it in the office or buying a new one. He can simply see what's available in the cloud and access it. If the right device is being used he can reserve it, and hold exclusive access to it for as long as he needs to test.
Cloud testing significantly reduces a mobile application's time to market by compressing the total testing time. A quality assurance manager can manage all the devices in the mobile lab by assigning projects and devices to different testers in different offices throughout the company. They can assign one task to his testers in Hong Kong. Once their shift is over he can assign the next steps of the project to his team in California. When their shift ends they can pass on rest of the work to the team in London. By navigating different time zones they can create a nonstop testing process where an application will be constantly tested until it is ready to be deployed to market.
The key is to having the right mobile cloud testing tool is security. An on premise device lab should operate protected inside a company's firewall, shuttling information from point to point safely within the company's VPN.
Break the Bottleneck with the Right Tool
The right set of testing tools can reduce the pressure to cut corners by empowering a business to test its mobile application thoroughly, and efficiently. If you can guarantee the functionality and performance of your mobile application, you automatically leap frog half your competition. Testing may not be the most glamorous part of the mobile development process, but the results of a good testing strategy are very glamorous indeed. A quality mobile testing plan is the hidden treasure to mobile success.
Experitest's SeeTest Suite meets all the necessary testing needs for today's mobile application.
- SeeTest Manual allows for manual testing using a mouse and keyboard. The product is free, so you can download it anytime and start testing immediately.
- SeeTest Automation speeds things up by allowing you to create scripts, and test them on multiple devices all at once. You can translate these scripts to any major testing language and integrate it into a CI process.
- SeeTest Network Virtualization simulates any local network, including creating special surprises like a user entering an elevator or riding a train through a tunnel.
- SeeTest Cloud lets you use all of these tools on devices to test on any device connected to your machine, and any device in your on premise device lab. All testing performed safely within your company firewall.
You can see a case study in this white paper on Mobile Banking Testing about how a large enterprise deploys the SeeTest suite of products (the case study is on page 29).
Using any of the SeeTest products, you can capture a device log using the getDeviceLog command. If you are looking to capture only certain parts of the log, you can use the SeeTest commands to clear the device log for customized tracking of events. To get the SeeTest application log, or a log of just what the testing tool is tracking on the device, you can use the getLog command.
SeeTest also enables you to monitor CPU, memory, and battery usage. With the getMonitors command you can test to see how your mobile application impacts battery and memory levels of the device itself.
Mobility is a new industry with opportunities and challenges we've never encountered before. The best tools are the ones that have adapted quickly to the changing environment and prepare you for the next leap forward in this rapidly expanding market.
About the Author
David Fink is a marketing manager at Experitest, a leading provider of mobile test automation, manual testing and monitoring tools for mobile applications. David worked on Wall Street during the internet boom of the 1990s. Watching the same transformation taking place in mobile today, he writes about the importance of enterprises to make sure their products are working perfectly so this era of prosperity can be even greater for them and their customers
