Formed in 2009, the Archive Team (not to be confused with the archive.org Archive-It Team) is a rogue archivist collective dedicated to saving copies of rapidly dying or deleted websites for the sake of history and digital heritage. The group is 100% composed of volunteers and interested parties, and has expanded into a large amount of related projects for saving online and digital history.
We’re all familiar with the story of the tortoise and the hare, where a plodding tortoise wins a race against a speedy hare because the over-confident hare takes a nap midway through the race.
It’s a valuable lesson about the power of smarts over sheer speed. But all due respect to the methodical approach, the tortoise does NOT win the race in mobile app testing. The hare wins. But he needs to be smarter than Aesop’s reckless rabbit.
Despite efforts to expedite app release cycles, enterprise mobile teams still struggle with what we call “velocity blockers”, which are testing practices and processes getting in the way of fast, iterative app releases. To get to the bottom of velocity blockers, the Perfecto Mobile research team surveyed a group of 350 enterprise-level developers, DevOps and QA people of various responsibilities about what testing-related issues are slowing them down.
The survey data revealed an encouraging shift to Agile practices that emphasize quick results and integrated feedback. It’s like having a hare’s body with a tortoise’s brain. Of our respondents, 86% said that they’re either Agile, or have made the first steps towards Agile. And 94% say they’ll be there within a year.
This shift is reassuring, but testing lags remain in the way of progress, according to survey results. Here are the five main velocity blockers cited by respondents, with tips on how to be smart like a tortoise while moving as fast as a hare.
- Slow QA to Developer Feedback
As developer teams begin using agile practices, the feedback loop between QA and developers tends to be unbalanced. Often when a company speeds up to a monthly release cycle, a team will spend three weeks developing and the last week testing and fixing bugs. The problem here is that as much work needs to happen in the final week as in the previous three weeks.
Solution: We recommend embedding quality throughout the entire lifecycle, a concept we call Continuous Quality. QA feedback is always visible and developers always know where their code stands because it’s constantly being tested. Maintaining this kind of control over the quality of your mobile app throughout all stages of your SDLC will boost the speed of development and reduce the number and severity of defects reaching production.
- Ineffective Test Automation
Automation, on its own, may speed up testing and releases. But it’s how automation is implemented that really makes a difference. Automating your test scripts is an important step, but it’s not enough. You must first make sure your environment is stable for automated tests. When you run automated testing across an unstable environment, false negatives happen.
Solution: Start by automating the provisioning of your test environment to make sure it’s stable and will support your test automation correctly. If your environment is stable, your tests won’t return false negatives.
- Delays in Device Readiness
Was your app ready to run on the Samsung Galaxy S6 when it released? Did someone on your team wait at the store to purchase it so you could deliver 0-day test results? Better yet, did you get it before it was released to test your app against it before users could upgrade? The answer is probably no, just based how quickly new devices are released to the market. But to maximize test coverage, device readiness is a full-time job. So how do you get the latest devices into your test lab, online and ready for testing before your users are impacted?
Solution: Let a managed lab-as-a-service do some work for you, where a vendor acquires new devices, maybe before they’re publically available. Let them manage your test lab for you, and dedicate experts to you. Make sure they also offer an SLA. It costs an estimated $6,500 per device per year to manage in-house. Multiply that number by the total devices you need to test on, and this gets unscalable, unmanageable and unprofitable.
- Slow Access to Mobile Devices
Let’s say you do buy devices and manage your own lab. The next issue you’ll run into is making those devices available to your testers all over the world in real time.
At one of the top U.S. mobile carriers, when a new device was released, the carrier had staff who had to acquire the device and ship it to where testing was performed. This group had limited time to test the device before shipping it to another testing group. When timelines weren’t being met, the staff member physically traveled with devices, from test group to test group, to ensure they made it into the right hands and on time. Travel costs made for an inefficient process.
Solution: Once the carrier chose to put devices in the cloud, the device procurement and deadline challenges were eliminated and a scalable, on-demand cloud emerged. They can now centrally manage devices for testing via the cloud and groups can access devices remotely while still ensuring the same user experience.
- Manual Testing Takes Too Long
Manual testing is by far the top app development velocity blocker. We work with a lot of the major banks because, more than most industries, banks win and lose customers on the merits of their mobile app. One of the top U.S. banks came to us back in 2010. They were doing lots of manual testing of a wide range of complex scripts. It took them a long time to push through their test cycles, and app defects were missed.
Solution: The bank’s pre-automated test environment relied heavily on manual testing, and included duplicate labs and a 16-20 day regression cycle. After automating their testing scenarios, they saw a 90% time reduction per regression cycle. That’s bringing weeks-long cycles down to mere hours. They were even able to quantify that they saved $40,000 per test cycle. In the end, the bank was able to decrease their cycle times and the defects released to production by doing test automation. A major velocity blocker was unblocked!
About the Author/Eran Kinsbruner
Eran Kinsbruner, is director of product marketing at PerfectoMobile, one of the leading mobile cloud and automation companies. Formerly CTO for mobile testing and Texas Instruments project manager at Matrix, Eran has been in testing since 1999 with experience that includes managing teams at Qulicke & Soffa, Sun Microsystems, General Electric, and NeuStar. The co-inventor of a test exclusion automated mechanism for mobile J2ME testing at Sun Microsystems, Eran has experience in the mobile testing world. You can find Eran on Facebook, Twitter @ek121268, LinkedIn, and his professional mobile testing blog at ek121268.wordpress.com.
