UX performance testing with GameBench and eggPlant

Editor's Notes

• #3: Thanks for tuning in. I’m going to run through a quick intro to GameBench and the metrics we reveal. Then we’re going to show a quick demo of how GB works with eggPlant. And finally I’m going to look at some examples of good and bad optimisation in retail apps, to show you the power of what GameBench reveals.
• #4: Customers include major hardware players like Samsung and ARM, QA houses like Enzyme, game developers like SpaceApe.
• #5: I used to use GameBench to help me with product reviews, but it was even a product. When I realised how important this sort of performance testing is, I quit journalism in order to work for GameBench.
• #6: GameBench was created to solve a market need -- games too often fail to perform consistently across the vast range of mobile hardware.
• #7: Even games from high-end, well-off studios like Rovio, using mainstream middleware like Unity, fail to run consistently -- even on top-end devices. Introduce the concepts of median and minimum frame rate.
• #8: Users notice these inconsistencies. It affects their reviews and hence the game’s commercial success.
• #9: Months of QA, soft launches, feedback, performance analysis, tweaking. Smaller studios can’t afford this, and therefore usually fail. The gap between the two is growing.
• #10: As genres get more crowded, the industry becomes ever more brutal. “Trusted” brands, known for quality, get discoverability in the stores. Indies often get swamped in a sea of similar-looking titles.
• #12: And this is what quality looks like, when you’re talking about the smoothness of a game. FPS is solid at 60fps during races. Very strong peak at 60fps on the stability chart. Remember this shape for later when we look at apps.
• #13: Well, that’s what we initially thought. That’s why we called ourselves GameBench.
• #14: We thought that games and apps were totally different when it comes to the importance of a visually smooth UX. But our assumptions were wrong.
• #15: ] We thought that games and apps were totally different when it comes to the importance of a visually smooth UX. But our assumptions were wrong.
• #16: Both iOS and Android operating systems run at 60fps, which means apps must scroll and animate at this rate too -- otherwise they feel janky. 60fps came from gaming, but is now a universal target on mobile.
• #17: PhoneArena tested frame rates of Android -- these are the frame rates an Android user will experience right before and after using an app. E.g. Swiping the home screens or accessing the app drawer both animate at 58fps, both on Samsung’s TouchWiz and raw Google. (If anything TouchWiz is slightly smoother today, as this table is a year old.) This tells us that app frame rates really need to be at least 55fps, and preferably above 58fps, if they’re to match the underlying OS.
• #18: We know that apps need to animate at or close to 60fps. So how do we test this for a specific app? The answer lies in a combination of GameBench, which measures frame rate and other key performance data, alongside eggPlant, which allows you repeatedly test user sessions automatically. We’ve now got a demo to show you what that looks like...
• #19: (*) Looking at the architecture of the solution. (*) Here we have eggPlant Functional driving and verifying an iPad as normal; and then uploading the results to eggPlant Manager for review. (*) But in addition we are now getting the UX and performance information from GameBench. (*) The GameBench agent runs on the same machine as eggPlant Functional; and eggPlant Functional controls it via the command line, i.e. starts and stops the profiling. So we have ePF running as normal, but it’s also getting extra profiling information by controlling the GB agent. (*) At the end of the test the standard ePF reports can be viewed in ePM, but now in addition we have full profiling and UX data available in GameBench. (*) So that’s it – very simple – and now lets look at that in action.
• #21: Let’s start with an example of a good app - ASOS for Android. It’s worth just looking at the basic shape of the chart: two sharp peaks, one at 0fps (representing static time) and one much bigger peak at 60fps (representing time spent at the target frame rate). That chart isn’t perfect -- a bit too much activity in the 45fps region, which drags the median fps down, but still… pretty smooth.
• #22: Two more smooth apps, again showing the same basic shape. A big, sharp peak at (or close to) 60fps, and a smaller sharp peak at 0fps. The relationship between the size of the two peaks is interesting -- an ebook reading app would have the sizes reversed, whereas in retail we’d expect a health shopping trip to mostly be spent on the move, so we can define an expected ratio between these peaks and look for exceptions to highlight problems. But we won’t get into that too much now.
• #23: Mango throws up a rather different chart. Both peaks are much broader. The 60fps peak isn’t at 60 at all, but at around 53fps, which is below our target. The 0fps peak is broader, showing more time spent on a roughly static screen. The ratios are out of balance.
• #24: We noticed the issues in the Mango app immediately -- the shaky, unresponsive scrolling, the slow load times. But so did other users, judging from reviews on the Play Store.
• #25: We can’t be certain that they’re describing frame rate issues, but these words -- “laggy” and “slow-motion” -- tend to be linked with this issue, rather than just with simple load times. It’s worth bearing in mind that Mango for iOS has good reviews, and shows a very n
• #27: Screenshots are captured once per second and allow problems to be correlated to onscreen events. Here’s the cause of the excess time spent close to zero: dead screens while the user is waiting for things to load. Other apps showed a spinning circle during loads, to tell the user to be patient -- something is happening. But Mango didn’t do that.
• #28: FPS is the starting point for any analysis of smoothness. You correlate to screenshots and then also to secondary metrics such as network traffic. Here, it looks like Mango is downloading big bursts of data at up to 30,000Kbps -- nearly 14MB in the space of 5 seconds, which might be the reason why images are slow to load.
• #29: By contrast, the iPhone version of the app has many smaller and slower bursts of traffic, which is something we tend to see on better optimised retail apps.
• #30: Here we’re looking at usage of the graphics processor, or GPU. It turns out that this is probably the biggest factor, since the GPU is getting maxed out for no good reason.
• #31: By now, we have a pretty good idea that somehow the layout of the app is highly inefficient. This is further corroborated by Android’s own “overdraw” feature, which shows a mass of pink on the Mango app (left), versus much less pink on the better-optimised H&M app on the right. It’s up to the developer to take it further and fix the problem.