While mobile is the biggest digital channel, and app stores are swamped with new apps—as well as iterations of existing ones—daily, the ability to release faster and with higher automation coverage is still a big pain for most organizations. In the recent world quality report by Sogeti, the findings were that only 29 percent of the tests are being automated.
When exploring the key reasons and challenges for such a low percentage, especially for the mobile space, we see five key challenges that can explain that low rate—some are very much related to the findings in the report.
1. Tighter release schedules leave less time to automate
Organizations don’t have time to integrate new tests into their existing test cycles, and therefore have less time to develop new test code.
Due to the complexity of the mobile environment, and time allocated for this task, there is less time to write stable test code that continuously runs in parallel across mobile platforms.
Continuous innovation drives test suite growth, making overall execution cycle lengthy for organizations who already have a working automation process. So, there are more new tests to write in less time. And every few weeks, new devices with new operating systems and new features (new sensors etc.) are released—which requires automation test maintenance and can reveal new defects.
2. Maximizing test automation coverage within an open source framework
A challenge for a high percentage of test automation is what capabilities test today’s automation frameworks can offer developers.
Open source struggles to keep up with new technologies such as image injection, face-ID and fingerprint authentication, and chatbots. This can force an increase in manual test case execution, and therefore increased manual testing debt.
Automating environments that have varying carrier network conditions, locations, background events, and other virtualized services is very difficult and requires skills
3. Test automation stability and reliability are a quality risk
The nature of mobile tests often pops false negatives due to environment, device stability as well as test specific issues. Dealing with the debugging of these tests is time consuming and adds risks to the overall test coverage.
In addition, lab availability for testing on both mobile devices, emulators and desktop browsers is an additional challenge that impacts automation stability, as well as adds to project delays.
4. Test automation flakiness and quality visibility introduces quality blind spots
The ability to optimize the tests during pipeline and across builds is a challenge, due to following aspects:
- Test suites include flaky tests that aren’t handled and are being carried from one cycle to the next.
- Tests are being duplicated by various developers without test suite governance.
- The test suite constantly grows and adds to the duration of the CI test cycle and regression cycles.
- Organizations are blind to key KPIs around their test automation suite quality: how many tests constantly fail, how long each test takes across builds, how many retries each test suffers from and more.
5. Test automation tool stack isn’t in sync with organizational capabilities causing testing inefficiencies
Each organization consists of various personas. Each has specific skills and strengths, and when the tool stack isn’t tailored to individual team members’ skills, the overall quality health for the product is being impacted. Tool selection is important and also needs to meet the testing practice that the team has adopted, whether it’s BDD, TDD, agile, or other.
Recommendations
To address these five challenges, teams need to operate with a valid strategy to select their test automation framework such that it meets the team members’ skill sets and software delivery practice—both developers and testers. When selecting a test framework, the team needs to assess the framework’s ability to support recommended practices such as Page Object Model (POM), integrate easily into tools used throughout the pipeline like GitHub, Jenkins, Slack, back-end services, and others.
Test automation suite maintenance over time is a common pitfall within organizations. Having the ability to continuously monitor and tune the test automation suite via reports and dashboards can help teams remain within test execution time boundaries, as well as provide them with the tools to extract flaky and problematic tests so the suite is as healthy as possible.