Performing mobile application testing can be a arduous task. It requires a methodical approach and lot of patience. Since there are many variables involved, it becomes even more difficult to keep track of each and every type of issue.
Mobile applications are key to success for many businesses and individuals. A successful app is always appreciated by users, however a lot of effort is put behind it to make it seamless and bug free.
Testing any type of mobile applications involve same amount of effort irrespective of operating system. Below are some of the tips that I learned hard way while creating my first few mobile device applications. You can use these as a starting point for testing Android, iOS, Windows or any other mobile application.
Have An Entry Criteria For Testing
As a mobile application tester, first thing to do will be making sure the application is ready to be tested. Many times developers are making changes to application and its not stable. Specially in initial stages of project. You should ensure following
Enforce Developers To Write Unit Test Cases
Ensure that developers provide you a relatively stable build of application to test. This requires developers to perform a basic level of testing before they just throw away the build. Many bugs can be easily caught during unit testing phases. Encourage developers to follow strong unit testing methods and ensure the code coverage is reasonable.
First step of automation should begin at the source code level. You must have developers write their unit test cases. Test driven development can reduce the bug identification lifecycle and many minor programming issues can be easily caught by thorough unit testing. There are dozens of unit testing tools and frameworks in all programming languages and it should not be difficult to start using. Unit test cases must be written by developers only.
Enforce Developers To Use Static Code Analyzers
There are many static code analyzers that can be installed by developers to ensure basic programming issues are not left before shipping a build to a mobile app tester.
Build Frequency
Ensure the builds are not too frequent. Limit the build frequency to a reasonable time so testing team can focus on testing more thoroughly.
Clear Communication : Features
Ensure you have clear communication between developers and testers. Testers must sk developers about the application feature and changes. Ensure testing team have enough information about current state of the mobile application build. e.g. Which feature is implemented and which is not so testers can spend time smartly on supported features and avoid anything that is not implemented.
Clear Communication : Application Versioning
Ensure the developers follow a consistent versioning of application. A issue or bug can be reported, however it can not be useful if you do not know which version of application had the issue. If changes are too frequent often the bugs are resolved even before it can be tested. Making sure the build version is accurate is really important part of testing process. Ideally every application build should generate a unique version number and should available in application to be seen by the tester.
Keep Sanity Test Cases Ready
Sanity test cases are absolutely must have. You should not proceed with full scenario testing without doing a quick sanity test on application. This will save you a lot of efforts and speed up things. In this test case you should keep all high level key functionalities with simple test cases that can be completed in few minutes.
Once all sanity test cases are passed, you should proceed with in depth scenario testing for application.
Automate
Automation is my favorite task in any application testing including mobile apps. Automating mundane tasks will save a lot of time and effort to the testers. It also makes communication easy since many common issues can be easily caught and automatically emailed. Even developers can run these automated test cases before sending it to Quality Assurance (QA) team.
Using Emulator can make the automation easier, however you may also need to explore options that can perform automated tests on real devices.
Test On Emulators Before Devices
All mobile application development platform support emulator or simulator to perform testing before application is installed on a real device. Emulator must be used to perform first round of testing. Re-Installing apps on device is time consuming and often unnecessary. Many basic issues can be easily caught in emulator based testing itself.
Testing On Device
Now comes the key to the mobile testing. The really important bugs are found during testing your application on a real device. Below guidelines should help you in device testing.
Testing One Device Is Not Enough : Test On Multiple Real Devices
Any mobile app must be testing on app devices that it supports. Many unforeseen issues are observed in device specific scenarios. This step become even more critical in operating system like Android where dozens of hardware vendors are creating different shape and size of devices that may behave very different from each other. Also there are some mobile carrier specific limitations on each type of device that limits certain types of functionality on few devices.
Test On Devices With Different Screen Size
Its important to ensure the look and feel of application is consistent on different screen sizes. There are many screen sizes available for same operating system including Android, iOS and Windows. Therefore a thorough UI testing needs to be performed to ensure consistency.
Test On Different Supported Version Of Operating Systems
There are many version of a operating system floating around. Not everyone has the latest operating system and many have very outdated or un-supported versions. Testing on all supported version is important at the same time knowing what versions are not supported will also help you set the user expectations. Many good applications receive bad reviews on app stores since they do not support a specific version of operating system.
The iOS users are mostly upgraded to latest version of operating system therefore it becomes little easier for tester since they have less version to test. On the other hand Android has dozens of versions floating around and many hardware vendors have customized their own versions as well. This makes it even more difficult since the tester needs to perform app testing on all supported versions.
Test On Rooted Devices
This may sound weird, however you must ensure that a mobile application is working as expected on a rooted device. Even If it is not supported than the expected behavior should be tested. Many people have rooted devices and you may or may not want to support it.
Many mobile carrier develop applications that are not supported on rooted device. They ensure it by first testing on some rooted devices.
Network Testing Requires Many Scenarios
Network testing is complex and consumes most of the mobile app testing effort. Any application using network for retrieving or sending data must perform these testing.
Test Network Application Without Any Network
Test the app with no network availability. The app should clearly communicate to user about network availability. Many applications fail to communicate and lead to user frustration.
You can simulate this type of testing by blocking the Wifi and any network signals using a blocker material (e.g. Aluminum Foil – http://www.amazon.com/dp/B00I9J3Q8W/ ) or metal box.
Test Network Application Without Cell Data Network
This scenario is simple, disable the cell date network and perform the required network operation on application. Observe the behavior. Some apps may use WIFI as an option, others may not support WIFI. Make sure the expected behavior is fully tested.
Test Network Application Without Wifi Network
If the application is using WIFI network than you must try to test it in absence of wifi network. Some of the scenario that must be tested are listed below
- Test application by turning off the WIFI in device.
- Test application by turning on the WIFI in device but keeping it out of range of any Wifi
- Test application by turning on the WIFI in device but keeping it at a place where Wifi signal is really weak.
- Test application by turning on the WIFI in device and moving it out of a Wifi network while doing the network related task.
Ensure Network Operations Are Performed In Background
All network related operations should non blocking on any mobile device. Ensure you are able to do other tasks when a network operation is performed. If a user needs to wait for a 2MB file download without being able to do anything else it is going to lead to bad experience and app ratings may go down.
Reinstallation Is Important
Test application re-installations is successful after every new version is created. This will ensure your upgrades will reach end customers.
Default Device Behavior Is Unchanged Within App
Test default functionality of device are not broken inside app. To do this make sure following things work as expected inside and outside app.
- If the device has a back button it should be functional same way its is outside the app.
- If device has a home button it should be functional same way its is outside the app.
- Application should not affect incoming call, messages or notifications default behavior while user is using it. This may be different if you application deals with specifically these functionality however most applications will not affect these functionality.
Test Application While In Motion
This test is important for any application using network based features. Well this sounds obvious however not many mobile application testers try to test applications while they are moving. May be go for a drive and use the application while car is in motion. (not while driving yourself of-course ). Perform a network operation and see how application behaves.
Look And Feel Is Underrated
Test consistent look and feel on all screens. The users may not be user interface design experts however they know what is inconsistent screen mean. User will loose trust in a poorly designed user interface application. Most popular application on all platforms are designed with beautiful and consistent user interface along with useful features.
Test Landscape And Portrait Mode For Full Functionality Separately
The user interface on landscape and portrait mode of a device needs to be fully tested for each screen and optimized for each mode. There is not one size fits all approach to it, however pay attention to the location of components and ensure your application is making best use of space on each mode.
Test Switch Between Landscape And Portrait Mode From All Screens
For every screen when a user switches from landscape mode to portrait mode or vice versa the transition should be smooth. All screens must be tested for the transitions both ways.
Keyboard Functionalities And Default Behaviors
Application should not alter the default keyboard behaviors.
- Ensure the enter button on keyboard submits the form.
- Ensure The keyboard is automatically popping on a text field focus.
Ensure The Type Of Keyboard Is Based On Type Of Field
Many different types of a keyboard layout can be shown to a user. Make sure the keyboard layout is smartly selected based on type of field.
- A number field may show the number keypad mode instead of alphabet mode.
- An email field may show keypad with @ symbol available on alphabet mode.
Application Performance And Reliability Testing : Crash Testing
Below are key things for Mobile Application Performance Testing. Some of the below scenario needs to be tested with mobile apps for checking behavior in exceptional conditions.
Test App On Low Battery
Test application while mobile battery is absolutely low level and going to go down. Observe application behavior during low batter and keep testing until and after mobile battery goes completely dead. Ensure application data or installation is not corrupted. This is best case for testing network related scenario as well.
Remove Device Battery While Application Is Being Used
Turn on a application and start an operation that takes few seconds. Remove the device battery while operation in progress. Observe the behavior of application and ensure no data is corrupted.
Enter Large Amount Of Data
Use application to send unreasonably long amount of data and observe it is able handle is elegantly. All text field should provide user feedback about limits of text size and in case too large text is entered it should be handled without application crashing.
Same scenario is also applicable for image, video or other file download or uploads. Ensure you know the size limits supported by application. Make sure to communicate it clearly on the app, otherwise it may lead to users frustration and bad reviews.
Understand When To Stop Testing
Testing can be exhausting and it may look like never ending task. Smart testers will know when to stop testing and let go of certain minor issues. Its important to release an application in time with minimum bugs, therefore you need to strike a balance between the two. Keeping a list of known issues in the application will help prioritize updates on application and keep users aware that some issues are present in application.