Mobile Testing, That's Just a Smaller Screen, Right? Stephen Janaway

Editor's Notes

• #2: Hi everyone. I’m Stephen and I’m going to talk about Mobile Testing.
• #3: I’ve been in the industry for 14 years. I have a website and blog Follow me on Twitter.
• #4: I also curate a regular podcast all about software testing. You can get more details at www.testinginthepub.com.
• #5: Here’s what I’m going to be talking about.
• #6: They can also be medical devices, smart readers, hand held scanners, and much more. Basically anything with limited memory, portable, preferably with a data connection.
• #7: Android Wear - http://googleblog.blogspot.co.uk/2014/03/sharing-whats-up-our-sleeve-android.html Android FIT APIs Wearable camera’s NEST thermostats
• #8: Remember when we had to carry all this stuff about? Now it’s just in one device. It’s taken less than 30 years to get to this point. Most people are carrying around a small mobile computer in their pocket.
• #9: It’s much more than just devices Imagine if something went wrong…
• #10: You are late for that important meeting because your phone alarm clock does not work. IPHONE EXAMPLE
• #11: You are lost due to a problem with your phone’s navigation software. IOS7 MAPS EXAMPLE. OK – so of course you could use a map but that won’t actually direct you. And maybe you don’t even have maps anymore.
• #12: Actual Hardware – drop it, get it wet, stuff is removable Consumer electronics – fast paced, device jealousy.
• #13: To people they are much more. Users are more emotionally attached to their mobiles than their PCs. PERCEPTION OF QUALITY Often a person’s phone is the first thing that they look at when they wake up. In fact it wakes them up!! HTC advert and personal experience. People classify themselves by what device they carry. It becomes part of them. The apps they use become status symbols. Think about users perceptions of quality on each device.
• #14: MOBILE USERS WANT A MOBILE OPTIMISED EXPERIENCE BUT THEY DON’T WANT LESS….
• #15: I JUST WANT TO POST A COMMENT. HEAR ME!! Mobile Web Mobile sales have already overtaken desktop sales, and mobile Internet usage is predicted to overtake desktop internet usage by 2014. It is only logical that mobile search will overtake desktop search at some point in the near future as well. Since 67 percent of users claim they are more likely to purchase from a mobile-friendly website
• #16: Mobile users have far higher expectations of mobile apps when compared to other software like browser applications. The majority of reports and surveys state that nearly 80% of users delete an app after using it for the first time! The top four reasons for deletion are always bad design and usability, loading time and crashes immediately after installation. PERCEPTION OF REQUIRED QUALITY IS HIGHER Nearly 60% of users will delete an app that requires registration, while more than half of users expect an app to launch in under two seconds. If the app needs more time it gets deleted. Again more than half of users experience crashes the very first time they start an app.
• #17: Apps stores enable this. Do you download apps with consistently bad review and 1 star? Often lots of other apps to choose from. YOUR BRAND IS YOUR BRAND, NO MATTER WHICH PLATFORM!!! Find out what your users want and understand more about your users via apps stores. So what’s important? Testing, at all levels of app design!! SO LET’S LOOK IN MORE DETAIL AT TESTING AN APPLICATION.
• #18: What makes mobile testing different? 
We will touch on these much more in the rest of the presentation.
• #19: So, as testers, what do we need to understand. Firstly...usability. Conduct real user tests. Observe users using the product. Make friends with your UX people. Use user profiles. Screen is smaller and used one handed. TEST LIKE THE USER DOES!!!! In app analytics. Website analytics. User persona’s. Similar applications.
• #20: So what are you looking for when you test for mobile? Consistent brand Engagement - People typically have more than one device. They get distracted.
• #21: You need to learn a whole new language when testing mobile.
• #22: Think about things like screen rotation, as well.
• #23: Mobile devices are low power, multi-tasking, lower performance and memory. Stress them. Test for the interactions.
• #24: Q to the class – what should we think about? Typically two processors – main plus GPU. Max 3GB RAM. 16/32GB ROM. Memory leaks, etc.
• #25: Users hate you wasting their data allowances. Test for it.
• #26: Being able to test effectively for mobile means understanding the mobile world in detail.
• #27: The number of mobile cellular subscriptions is about to equal the number of people. Last time I looked it was - 7,259,491
• #28: Android is king. Note Series 40 which MS just killed.
• #29: Interesting thing here is the leader is the Android browser worldwide NOT Chrome as shipped on most recent Android devices. AOSP… UC Browser is a 3rd party browser which uses compression technology.
• #30: So here’s what I think we’ll be doing in the future of mobile testing.
• #31: Here’s the challenge – how to ensure testing covers enough of them? How do you keep on top? More wearables means more devices to get. More complicated test labs. More interactions between devices. More difficult test scenarios. IT MEANS USERS ARE BECOMING MULTI DEVICE AND THEREFORE MORE DISTRACTED AND MORE TIME POOR. Apps need to really grab the attention.
• #33: Tricky – lots of versions in live.
• #34: And it’s also about screen sizes. And carrier software.
• #35: Subscribe to the relevant updates/ seed notices from platform manufacturers Make friends with UX
• #37: So what should you do? Not just simulators: They don’t fully emulate the device (some are better than others). They are not running on low performance hardware. Cannot use on real network connections. Security is normally permissive. Just be aware that cloud based services have some issues with latency, and try them out first to make sure you are happy using them.
• #38: A great idea. Why not start one?
• #40: As testers – start taking control of this.
• #41: Users still don’t trust mobile as much as desktop for commerce. Browse on mobile, buy on desktop. But this is slowly changing with things like Paypal and google wallet. APPLE PAY. Barcode scanning, contactless payments, london tube, location based use cases. Security is becoming more and more important. What does this mean for testing?
• #42: Test for security issues. Think about things like browser security, cookies, etc. Security settings. App permissions.
• #43: It’s your companies brand, no matter which platform. As testers we need to ensure companies are taking security on mobile much more seriously. User and credit card data. Stored on device or server? Stored encrypted? Stored on SD or SIM card? Removed when the app is uninstalled? Security of channel. Wifi sniffing. Man-in-the-middle. User left logged in. Can the app be de-compiled? MobiSec and OWASP.
• #44: And it’s not only real security issues. Perception of security. Application permissions. Watch out for these. Are they appropriate? This is a flashlight app (malicious example but sometimes it’s just due to lazy coding). TEST FOR THIS BEFORE LAUNCH. Users perception of security (affects mobile web too).
• #45: What does this mean for testing? Localisation Different character sets Right left languages Is the app available in different languages? What happens if you change languages? What happens if you change keyboard language? Watch out for right-to-left languages – a frequent source of UI bugs.
• #47: Wearables drive this.
• #48: So – for the future why not make a commitment to avoid some of the common Mobile Testing Gotcha’s?
• #49: Mistake number 1 As we mentioned before: They don’t fully emulate the device (some are better than others). They are not running on low performance hardware. Cannot use on real network connections. Security is normally permissive.
• #50: Mistake number 2 A mobile device lab is not just for Christmas! This is consumer electronics. You need to keep it updated regularly. Think about open device labs as well.
• #51: Mistake number 3 Users are not in your office!!!
• #52: Don’t just test in perfect Wifi and perfect light. Test in low signal conditions. Opensignal.com is brilliant for this.
• #53: Test on the train. Overloaded networks. – test in busy areas. Waterloo station is the rush hour for example  Test for swapping between 2G/ 3G/ Wifi. Speed.
• #54: Deadspots Mention testers knowing where all the deadspots in Nokia offices are. Find a walking route with poor connections and record it using a fitness app.
• #56: Mistake number 4!!
• #57: Applications with poor battery life are hated by users. Moves being a good example. Some users just blame the device. Battery life – typically a problem with apps that use advertising incorrectly (Android especially). Very, very important. Typical scenarios: Normal Low battery Battery drain time Affect of poor/ no network connectivity. Does the app keep trying to connect?
• #58: Mistake number 5 Ignoring OS updates or thinking that they don’t affect your app.
• #59: iOS and Android (to Google Play devices), happen quickly. For the rest carriers control things and their acceptance testing adds significant delay. Always test before the update. Get on the developer programs and get early versions of the OS. But beware – they are not always the final functionality so be sure to also regression test in live after official update.
• #60: Mistake number 6 Mobile test automation is like desktop test automation 6 years ago. You may want to consider pushing for more unit and integration tests before spending time on mobile UI automation.
• #63: So – if I could sum up a mobile test strategy in 5 top tips, what would it be?
• #64: And I’ll leave you with a quote from someone I used to work with in Nokia, which sums up why I think mobile and mobile testing is cool.
• #65: As a project manager once said to me in Nokia: “Be proud, your work will soon be in the hands of millions”. No pressure on us testers, right? 
• #66: Please ask any questions. REMEMBER TO REPEAT THE QUESTION BACK TO THE AUDIENCE BEFORE ANSWERING. I’m about for the rest of the conference so please come and have a chat. Or you can contact me through my website or via Twitter.