Overcoming the Challenge of Mobile App Testing
Smart-phones are becoming more prevalent by the day and that day is not far when they will be our primary access to Internet. Most software products or web applications now come with their mobile versions. The remaining (if any) will soon follow suit. As of April 2012, there were more than 1 million apps on the 2 leading app stores combined (Apple and Google). Businesses are realizing that mobile is the best way to reach maximum customers. Hence E-Commerce apps are burgeoning. As apps are rising in value chain, the need for quality testing these apps is also increasing. There are several challenges in mobile application testing though. Let us look at some of these challenges and also the ways in which a tester can encounter them.
1. Multiple OS Platforms and Fragmentation
This is the biggest cost factor in any mobile application. Ideally, businesses would like to reach the maximum customers. Hence they'd like their app to work on all OS platforms. But testing the complete functionality on multiple platforms is expensive. Hence the challenge is to find a sweet spot between too much testing and too less testing. So, let us see some strategies to find this sweet spot.
a. Create the following test suites:
i. Smoke Test Suite - The most critical test cases that need to pass on each platform.
ii. Regression Test Suite - Test cases that cover all the critical features that need to be tested with every build, but can be executed on any 1 platform.
iii. Integration Test Suite - Create a Test suite for testing Native feature integrations like Camera, GPS, accelerator, sharing, etc. These test cases need to be tested on every platform that your app supports.
iv. Current Release Functional Test Suite for features that are going into the current build/ release.
b. The Smoke Test Suite needs to be executed on all platforms whenever there is a new build or release.
c. Distribute all your Regression test cases across all platforms. This way, executing them only once gives you coverage on all platforms. For the next release or execution, change the distribution such that after a few executions, you would cover the regression test cases on all platforms.
d. The Current Release Functional Test suite needs to be executed on all platforms (emulator or hardware), but need not be executed on all fragments of the OS. Instead a distribution strategy like above can be used.
2. Emulator or Actual Device
A mobile application depends heavily on Native hardware integrations. E.g. integration with the Camera, GPS, accelerometer, Mobile data, Bluetooth, NFC, etc. So, let us see some strategies to efficiently control the cost of buying multiple hardware.
a. If these integrations are used for critical features of your application, having the hardware to test those is essential. You cannot rely on the emulator to mock the hardware interfaces. But, there are a few things to consider when buying hardware.
i. Do not buy hardware for each version of the operating system, since an OS interface with the hardware does not change with every OS version. Find out if it is really required before buying it. Also, this is a sure shot way to bankruptcy
ii. Try to use a Virtualization solution which provides virtual access to actual hardware. One such service is http://www.perfectomobile.com/ and another onehttp://www.keynotedeviceanywhere.com
b. But if your application is not dependent on these integrations heavily, use emulators as much as possible. They are faster to set up and test on. There are multiple emulations available for sensors like GPS, accelerometer, etc specific to the platform. Some of the features that can be easily tested on emulator are:
i. Different Screen sizes and resolutions.
ii. Mocking GPS Coordinates (For testing applications offshore).
3. To Automate or not
Mobile applications are typical small. But just because of the multiple platform possibilities, the potential of retesting and regression testing is huge. This is why automation for a mobile application is critical for longer duration projects. Let us see how we can leverage Automation for the best ROI:
a. Automate the smoke test suite as much as possible.
b. UI Automation is time consuming and fragile, hence try to use a good automation framework like Keyword driven or Hybrid framework.
c. Automate most of the functionality using web services or APIs for faster and robust scripts.
4. How to test Usability
Usability can make or break a mobile app. Testers are in the best position to experience the usability constraints since they are aware of the end to end flows and spend most time with the application. Even if it is not an explicit requirement, a good tester should look for usability issues and report them. This is where the tester can provide a value add to the team. To get better at finding usability issues, do the following:
a. Research the market (Application Stores) for similar apps and study them. See how they are doing things. You may learn a thing or two from them.
b. Compare the app with the PC version. Users usually try to find the functionality available on PC software in a mobile app. See if the user can find the critical features easily.
c. Review the results if any analytics was done on the PC app. This will give you a hint on the most sought after areas of the app and test them for user experience.
d. If testing a native app, keep updated with the latest OS recommendations for native feature integrations like accessing the phonebook, menu styles, breadcrumbs, etc. Report issues where your app does not meet the standards.
e. You may have a separate performance test effort running, but a good tester should report any performance slowness or battery consumption issues that are apparent during the functional testing.
f. Plan for some time to perform just exploratory testing.
Mobile testing is seemingly easy, but a tester needs to be more aware when testing on a mobile device. It is easy to ignore minor travails and use workarounds in the test environment as the app is under development. But this could become a habit and ignored when the final app is ready. Hence even when minor deviations are found, it is critical to document them (this need not be a detailed bug report, but a simple note to check later). Equally important is to have a list of planned test cases that are categorized in suites and executed before a release.
1. Multiple OS Platforms and Fragmentation
This is the biggest cost factor in any mobile application. Ideally, businesses would like to reach the maximum customers. Hence they'd like their app to work on all OS platforms. But testing the complete functionality on multiple platforms is expensive. Hence the challenge is to find a sweet spot between too much testing and too less testing. So, let us see some strategies to find this sweet spot.
a. Create the following test suites:
i. Smoke Test Suite - The most critical test cases that need to pass on each platform.
ii. Regression Test Suite - Test cases that cover all the critical features that need to be tested with every build, but can be executed on any 1 platform.
iii. Integration Test Suite - Create a Test suite for testing Native feature integrations like Camera, GPS, accelerator, sharing, etc. These test cases need to be tested on every platform that your app supports.
iv. Current Release Functional Test Suite for features that are going into the current build/ release.
b. The Smoke Test Suite needs to be executed on all platforms whenever there is a new build or release.
c. Distribute all your Regression test cases across all platforms. This way, executing them only once gives you coverage on all platforms. For the next release or execution, change the distribution such that after a few executions, you would cover the regression test cases on all platforms.
d. The Current Release Functional Test suite needs to be executed on all platforms (emulator or hardware), but need not be executed on all fragments of the OS. Instead a distribution strategy like above can be used.
2. Emulator or Actual Device
A mobile application depends heavily on Native hardware integrations. E.g. integration with the Camera, GPS, accelerometer, Mobile data, Bluetooth, NFC, etc. So, let us see some strategies to efficiently control the cost of buying multiple hardware.
a. If these integrations are used for critical features of your application, having the hardware to test those is essential. You cannot rely on the emulator to mock the hardware interfaces. But, there are a few things to consider when buying hardware.
i. Do not buy hardware for each version of the operating system, since an OS interface with the hardware does not change with every OS version. Find out if it is really required before buying it. Also, this is a sure shot way to bankruptcy
ii. Try to use a Virtualization solution which provides virtual access to actual hardware. One such service is http://www.perfectomobile.com/ and another onehttp://www.keynotedeviceanywhere.com
b. But if your application is not dependent on these integrations heavily, use emulators as much as possible. They are faster to set up and test on. There are multiple emulations available for sensors like GPS, accelerometer, etc specific to the platform. Some of the features that can be easily tested on emulator are:
i. Different Screen sizes and resolutions.
ii. Mocking GPS Coordinates (For testing applications offshore).
3. To Automate or not
Mobile applications are typical small. But just because of the multiple platform possibilities, the potential of retesting and regression testing is huge. This is why automation for a mobile application is critical for longer duration projects. Let us see how we can leverage Automation for the best ROI:
a. Automate the smoke test suite as much as possible.
b. UI Automation is time consuming and fragile, hence try to use a good automation framework like Keyword driven or Hybrid framework.
c. Automate most of the functionality using web services or APIs for faster and robust scripts.
4. How to test Usability
Usability can make or break a mobile app. Testers are in the best position to experience the usability constraints since they are aware of the end to end flows and spend most time with the application. Even if it is not an explicit requirement, a good tester should look for usability issues and report them. This is where the tester can provide a value add to the team. To get better at finding usability issues, do the following:
a. Research the market (Application Stores) for similar apps and study them. See how they are doing things. You may learn a thing or two from them.
b. Compare the app with the PC version. Users usually try to find the functionality available on PC software in a mobile app. See if the user can find the critical features easily.
c. Review the results if any analytics was done on the PC app. This will give you a hint on the most sought after areas of the app and test them for user experience.
d. If testing a native app, keep updated with the latest OS recommendations for native feature integrations like accessing the phonebook, menu styles, breadcrumbs, etc. Report issues where your app does not meet the standards.
e. You may have a separate performance test effort running, but a good tester should report any performance slowness or battery consumption issues that are apparent during the functional testing.
f. Plan for some time to perform just exploratory testing.
Mobile testing is seemingly easy, but a tester needs to be more aware when testing on a mobile device. It is easy to ignore minor travails and use workarounds in the test environment as the app is under development. But this could become a habit and ignored when the final app is ready. Hence even when minor deviations are found, it is critical to document them (this need not be a detailed bug report, but a simple note to check later). Equally important is to have a list of planned test cases that are categorized in suites and executed before a release.
Posts
2 comments:
When testing using the traditional method on windows/web applications, we all know that we need to test functionality. The same concepts apply for Mobile Application Testing for smart phones as well. As testing experts we should ensure the functionally of the application. your content great help provide for mobile application testing thank you sharing a good content nice job keep it up
This post is very useful and its worth reading. If you wish to get services for mobile app testing then you can visit salvusappsolutions.com
Post a Comment