Introduction
In an economy where apps have become the very heart and soul of almost any business, you have less than one second to impress your user. Because of this limited impression availability, application performance is essential to ensure the quality of your customer's digital experience and your user loyalty.
Application Performance Management tools and methods are indispensable in ensuring application performance in real, live environments. Testing application performance should be initiated as early as possible in the application development lifecycle to avoid poor performance and ensure user customer retention.
To comprehensively assure performance quality, a holistic layered approach should be used to integrate protocol level performance testing - traditional performance and load testing with app level performance testing and monitoring of usage patterns.
-
Monitor/ investigate app user patterns
There are multiple solutions (usually SDKs, open source and commercial) today that enable tracking usage patterns in-app, these stats can, when collected with load and without load, can be used to gain a better understanding of the stability of the app The sort of answers that these solutions give the developer are responses to the following:
- What is the most commonly used screen in the app?
- What is the most used area in the screen?
- Do the users require a different layout of controls to use the app?
- Where does the app crash the most?
- How does your responsive website render on different devices?
- Where does the app respond with the slowest speed?
- How long does it take to load data into the app?
- How long does it take to load a page into a hybrid app?
- How does the latest deployment/ build compare with previous build responses?
- Does the app behave differently or slower while backend performances are occurring?
Some solutions can easily collect the above information from the app and aggregate the results from thousands of real users. This tip highly correlates with tip Beta management tip.
-
Test functional scenarios with different connection qualities/ Virtual networks simulation
Sending traffic over different Internet backbones and testing your app with different connection qualities and types, cellular operators and locations, will give you the confidence that the application performs consistently and reliably across a spectrum of infrastructures. Additionally you get to see the app performing when the user changes from an open network e.g. 3G/4G to a private network e.g. secure WiFi or LAN.
-
Test in production environment
Variables such as firewalls, specific load balancing configurations, server’s memory configuration, etc., may severely influence on performance. It is best to test in production or simulated production. This will tend to be done in a non-destructive manner. But the principle is consistent, test everything both physical, hardware and network to a full picture of what happens when the app is launched and used.
-
Functional test while load is occurring/ The Importance of Hybrid Load
Imagine that a functional defect can only occur under a specific load; it would be extremely difficult to find and reproduce that defect. Functional testing during protocol level load testing (hybrid load) also helps to measure and assess the user experience during different load profiles. The load scenarios are functional in their essence but they are not checked on the GUI level and don’t vary. As such the complementary functional testing that is suggested here is not limited to the scenarios that are simulated in the load. Again the move is to test as much as you can before you go live, this reduces the risk and increases the chance of the app being the success that is hoped for.
-
Evolution of Performance testing to System Performance Monitoring
Leverage your performance scripts to run constantly. With proper correlations, you will be able to drive business intelligence, such as impact of poor or good performance on revenues and the impact of different usage scenarios on your system. The reports you will get from your load testing tool will help you to get full understanding of the important questions, such as “what, when, why”.
-
Conduct on-going performance sessions/ embed performance testing into CI
After gaining confidence on the performance of the application through a time restricted performance testing sessions with post-session restarts, test the system from a performance point of view for a longer period of time. This will give you an idea to see how it survives time-endured sessions and recovers and ramps up from a stressed session to an ideal state. This is especially relevant when considering how users behave using apps on tablets, multi-tasking and switching between apps in short bursts.
-
Consider Beta-Testing/ Beta Management
Release management is a critical path in deploying new versions of your mobile application. Why not launch your new functionality to a limited crowd, let’s say 100 if your organization employees and another 100 that are your customers? Consider what crowd source companies have to offer. A controlled beta management that also helps you to control the release gather the outcomes – i.e. defects and delete/ disable the app upon request, will make a massive difference to the long term success of your application. The creation and execution of load sessions for your beta users is another level of assurance that can be gained from adopting this discipline.
-
Backend Database Testing/ Monitoring
If you ignore the performance of the database that drives your application, your end user's experience will suffer. Setting and monitoring performance baselines for your database (as a standalone unit under test and as part of the system) through roll out is absolutely essential. This may include running production system stress tests to ensure your database can handle the new data loads, setting thresholds to avoid inefficient or poorly performing queries, and tracking real user response times to ensure a consistent user experience throughout the roll-out process. It’s obvious to take an end-to-end approach but spelling out the consistent elements that need to be testing will ensure you have covered all of the bases.
-
Perform Log Analysis – App and Client Side
Log files contain valuable information on the performance of your application. Not all problems are visible and users, and not all users have the ability to understand the root-cause of the problems, especially the non-reproducible. Log Analysis tools let you monitor servers and services, generate usage and error reports on the client side, build compliance analytics and more. Collecting log data from servers, networks, devices and security systems are all part of giving you as full a picture as possible of what is going on when an application is launched and used. Log analysis should be deployed during development, functional testing, load testing and production. Better to get a comprehensive understanding of error logs before users get them and stop using an application that you have spent so long (and much) developing.
-
Involve DevOps
Get your Operations Team involved. Development and Operation departments view application performance differently, largely because it is a concept that consists of multiple complementary approaches for addressing issues surrounding application performance. Understanding the different requirements for Dev and Ops is important to ensure app performance before a rollout. Once the requirements are defined from both teams, build an application performance baseline before the go-live date. This requires an understanding of the application and knowledge of how the metrics are being collected. It’s easy to overlook the fact that an application impacts both external audiences (user experience) and internal stakeholders (operations). DevOps might be a buzzword, but ignore all of your key stakeholders at your peril.
-
User Experience
User Experience cannot be properly evaluated by your testers or developers. Their knowledge of the application creates an unintentional blindness that prevents them from noticing problems. User experience is best evaluated by real users, an external user experience expert or a combination of both. Certain response time of application behavior may look normal for one user but will encourage other users to uninstall it. Ideally use as many external people (i.e. crowd testing) in your performance testing as possible so you can perform feedback reviews to understand their opinion about the app. The more you do up front the better it will be long term as the saying goes ‘plan ahead to avoid disappointment’.
-
Crowd Testing
Crowd Testing allows you to gain the benefit of different mindsets, handsets, geographies, configurations and more. Run a load session where you ask several hundreds, or even thousands, of real users to use your application in certain load time slots. Incentivize the users to report problems to you – either by dedicated communities, crowd sourcing companies or by social media. Crowd Testing is about as close to real life usage as you are going to get and can be a type of hybrid load.
Application Economy Growth
As of October 2014, Tech Crunch reported that the total app store revenue for Apple was $85 billion dollars1, which does not even include the billions of dollars of revenue on other smartphone platforms. It is imperative that the digital approach of both web and mobile applications meet the performance, scalability and quality expectations of users; otherwise, the effects on your organization can be devastating. Carrying out the tips above will ensure your app is a revenue generator and helps move your organisation forward rather than another sorry IT implementation failure.
Citations
1 Perez, Sarah. "App Store Downloads Top 85 Billion, Revenue Up 36 Percent Year-Over-Year." TechCrunch. Tech Crunch, 20 Oct. 2014. Web. 17 Nov. 2014.
About the Authors
Aviram Shotten QualiTest UK Vice President for Professional Services, has over 12 years’ experience in SW Quality, System Engineering and process improvement. Since joining QualiTest in 2005 he has led some of QualiTest’s strategic customers around Medical Devices, Aerospace and Defence, Retail and Media and professional practices around test automation, performance testing, user experience and Crowd sourcing.
Ayal Zylberman is President, QualiTest. He is a veteran in the field of software testing, specializing in various industries such as military, billing systems, network management, networking, and telephony since 1995. A pioneer in testing automation, Mr. Zylberman was among the first to implement Mercury’s automated tools. Mr. Zylberman is committed to spreading global practices, such as Keyword Driven Testing, Scrum Based Testing, and more, among QA teams in leading organizations. With numerous published articles and an active lecture schedule, Mr. Zylberman is recognized as an expert in the field of quality assurance and testing services worldwide. Mr. Zylberman holds a BA degree in Accounting and Economics from the Tel-Aviv University, Israel and an MA degree in Economics from Tel-Aviv University, Israel.
