Back in 2014, game developer Red Winter had to pull the sequel to its hit game Dungelot from the app store, mere days after first releasing it. The reason? The game was riddled with bugs, and it had switched from a pay-once-play-forever model to a freemium model.
Many of these issues could have been avoided had Red Winter followed a more thorough beta testing strategy. And this is something that doesn't only apply to mobile games, but all mobile apps.
When launching an app that is too buggy, or includes features that are unpopular with many users, you risk permanently damaging the reputation of your app. Red Winter was able to fix Dungelot 2 and relaunched it more successfully three months later, but there is no guarantee that your app will recover as easily.
Beta testing your app is a very important step in the pre-launch stage of your app development since it could highlight any problems with the following:
-
Quality: You may be sure that you have built a quality app, but only with proper beta testing will you be able to see that all the features function the way they are meant to. Quality is closely linked to the next point,
-
Usability: From UI through to UX. A usable app is one with an intuitive user-interface, with users easily able to navigate through your app, find what they want, and do what they expect to with your app. You are looking at how they perform certain functions, and seeing if there are any ways of improving the flow.
-
Bugs: Naturally you would have been thorough in ensuring there are no bugs in your app, but until it is used in a real-world scenario, you can't be sure. From serious bugs that cause the app to crash, to minor bugs that only reveal themselves under certain conditions.
-
Performance: The device, operating system, and even other apps could all affect the speed and overall performance of your app. Beta testing gives you an opportunity to analyse this more authentically than your own lab tests.
-
Marketing: From word-of-mouth marketing between your testers and their friends, through to insights into ideal audience as revealed by your testers, beta testing can help shape your marketing strategy pre-launch.
After considering these points, you'd find it difficult to deny that beta testing could help your app, and its launch, tremendously. And setting up beta testing isn't a complicated process, with a multitude of tools available. Some of these tools focus on just one aspect of beta testing, while others try to offer a more comprehensive set of features. Here's a rundown of some of the best ones.
UXCam
Discussions around UI and UX entered the mainstream in the late '90s as the internet became more accessible and more popular. And while the first mobile apps might not have made good use of UI and UX -- after all, it was a brand new field -- most large app developers now have separate departments focused only on UI and UX.
Strictly speaking, UXCam isn't a beta testing tool. It integrates into your app (iOS or Android) through the addition of two lines of code, so you can either limit its use to your actual beta testing or continuously in the final release of your app.
As the name suggests, the tool helps you assess your app's UX by making a screen recording of all user actions in the app, including touch points and gestures. The reason you would want this is because actually seeing how users use your app is a lot more insightful than regular user feedback and standard app analytics.
Additional features include a heat map -- highlighting what areas of the screen users interact with frequently, and which areas they ignore. The user flow feature is similar to the one found in Google Analytics and shows how users navigate through your app, and where they drop off. And with smart filtering, you are able to easily filter recordings according to the criteria you want to focus on, such as users who have been using your app since a specific date, or even the number of gestures used.
UXCam is unobtrusive, and your users won't even be aware of it. This, along with the fact that it doesn't require any user permission, could raise concerns about privacy, but UXCam has a strict policy requiring all sensitive information to be blocked.
Plans for UXCam include a limited Free plan, with the Pro plan (ideal for small businesses) starting at $199 /mo.
UserTesting
At first glance, UserTesting is very similar to UXCam. It gives you access to videos (and audio) of real-world users interacting with your app. However, unlike UXCam, UserTesting seems more like a beta testing tool in that it can be used at any stage in your development cycle, and you specify which tasks you want to test. So instead of seeing how users interact with your app as a whole, you can more narrowly focus on specific sections of your app, and specific tasks.
UserTesting doesn't only give you a video recording of actions taken, but also audio of users discussing your app. Additionally, you can present users with a series of questions, and get them to rate certain features. And results are available within an hour of each test being run.
Although you are using testers from UserTesting's own pool of users, you are able to specify the type of user you are targeting - from region, through to web expertise and other key demographics. And running new tests is as simple as uploading your app to the UserTesting platform -- no SDK integration needed -- and then creating the test parameters.
And if you're still unfamiliar with beta testing, the UserTesting website includes rich resources, including handy checklists for you to follow. The Individual plan is quite sufficient for independent developers and small businesses, with an Enterprise plan offering more advanced features.
99tests
99tests is promoted as a crowdsourced testing platform. What this means is that, like UserTesting, you specify the audience type -- key demographics, device type, location, etc. -- and 99tests find the perfect testers for you.
As with other testing tools, usability testing is offered, but 99tests also offers localization testing, exploratory testing, and security testing. Localizsation testing is testing your app among users in the locations your app will be available in, taking into account how local users respond to the design, etc. Most test cases call for users to perform certain actions, and follow a predefined path through your app, but exploratory testing frees testers up to explore your app without restrictions. You have a preconceived idea of how you want users to interact with your app, but you don't actually know whether they will. Exploratory testing could help you identify problems that only manifest when not using the app the way you design it.
And of course, security testing is critical if your app requires registration or the ability for users to input sensitive information. Data leaks are a nightmare to manage, even for large corporations.
Another feature that sets 99tests apart is that there is an element of automation to the testing. You are expected to compile a test plan for each round of testing, but the 99tests testers are able to create their own test cases, with automated scripts allowing them to test dozens of different scenarios in a very short period. And since 99tests integrates with many popular bug tracking tools, keeping track of reported bugs is managed through a single dashboard, with an app grader score helping you keep track of the apps perceived quality after each test.
Pricing for automation, risk coverage, and enterprise testing are available on request, with a quick 'bug bash' plan starting at $500. Most testing is run over a month long period, with complex apps needing 2-3 months.
TestFairy
TestFairy is another beta testing tool that provides a video recording of what users are doing with your app. However, TestFairy does not give you easy access to a pool of testers, and you are expected to source your own testers. This shouldn't be a deal breaker when considering beta testing tools, but it does mean you should have a tester 'recruitment' plan in place before signing up for TestFairy, if you don't already have your own pool of testers. Using family and friends is a good start, but only if they match the demographics of your intended audience. You want a diverse pool of testers who are able to test your app on different devices, and in different real-world situations -- for example, using WiFi versus mobile data.
TestFairy supports testing of both native apps, and apps developed using platforms such as Adobe Air, Unity, etc. And if your testing plan is focused on finding -- and eliminating -- bugs, you'll be happy to know that TestFairy integrates with several top bug tracking services . And your users only need to shake their device to call up the bug reporting form!
TestFairy is also very aware of how fragmented the mobile world is, so they give you the opportunity to see what your app looks like on more than 10,000 devices. It sounds exhausting, but if you're trying to ensure your app works perfectly for the largest possible audience, it is small sacrifice that needs to be made.
Plans for TestFairy start at no cost (as in 100% free), making it an ideal solution for SMBs with a limited budget. If you have a large team working on your app then you may want to consider their startup and enterprise plans.
HockeyApp
The final tool in this roundup of top beta testing tools for your app is one that offers more than just a testing tool. HockeyApp offers the standard features you would expect in a beta testing tool, but they also include user metrics. User metrics in HockeyApp encompass everything from number of active users and engagement, through to information on devices on which your app crashed in the last 30-days. This last metric can be quite valuable in helping you differentiate between bugs that affect all users, and those that affect users on specific devices.
Like TestFairy, HockeyApp can be used for both native apps, and apps developed using other platforms. And while any crash reports are by default sent directly to the HockeyApp platform, you can also integrate these directly with your preferred bug tracking platform. And communication among team members can be limited to the HockeyApp dashboard, or integrated into Slack and HipChat.
HockeyApp was acquired by Microsoft in 2014, but this does not affect your ability to test both iOS and Android apps, while the platform as a whole is busy migrating to a Visual Studio Mobile Centre, which new users can sign up for access to now already. Again, as with the other tools discussed here, integrating HockeyApp into your app is quite simple, and the SDK can either be integrated manually, or by using one of the client apps provided by HockeyApp.
Pricing for HockeyApp is also quite simple. Developers with fewer than 10 apps can use the service for free, with an unlimited number of testers for each app. Pricing after this is tiered according to the number of apps you have, with plans starting at $30 /mo for up to 15 apps.
Playing both sides of the mobile fence
There are many tools available for beta testing your mobile app, but I have selected these because they all support testing iOS and Android apps. In the early days of the mobile ecosystem it was possible to only target iOS or Android, but this is no longer viable. And even though Android has a significant lead over iOS, focusing on one operating system only still limits your reach. Additionally, this selection of tools includes options for developers who only want to focus on testing their UX, and those looking for a more comprehensive testing suite.
Regardless of the potential size of your audience, opening your app to beta testing shouldn't be something you have to think over. Your customers will enjoy a more polished product, and you will enjoy more loyal users.