Mobile Website Optimization
- 1. The Magazine for Professional Testers March, 2010 ISSN1866-5705 www.testingexperience.com freedigitalversion printversion8,00€ printedinGermany Advanced Testing Techniques 9 © iStockphoto.com/NWphotoguy ConferencesSpecial
- 2. 90 The Magazine for Professional Testers www.testingexperience.com Mobile Website Optimization using Multivariate Testing by Yeshwanta Hebalkar © iStockphoto.com/HelleM It was very frustrating just a decade and a half ago to establish net connectivity for offices in a small town in India away from the hustle and bustle of the typical busy metro. The objective was to establish dial-up connectivity to the National Informatics Cen- ter’s district office in the town. It was a lot of blood & sweat with typical text command prompts using UUCP connection and mo- dems buzzing in in the background. Incidentally, at that time the telecom boom was about sending SMSs using pagers. Today, with the exponential growth of net and associated digital technolo- gies, its penetration into our lives is mind-boggling. Proliferation of mobile devices has created a phenomenal impact on our lives due to their multi-functional potential and reach. The convergence of computing, net with mobile telephony servic- es and its evolution is amazing, leading to the smartphone gen- eration. Typically, smartphone features are touch screen, GPS, ac- celerometer, camera, net, and keypad. Refer to the recent launch of Google’s Nexus One. It has voice support for all text fields. You can now dictate your Tweets, Facebook Updates, and emails. Among the smartphones, typically Apple’s iPhone is considered as the game changer. This is because it has addressed many of the problems of the mobile ecosystem from how people interact with the phones, to where they buy phones, to what type of apps users will pay money for, to level of tech standards we can support on constrained devices. Apple has done it and others such as Nokia, Blackberry, Microsoft and Google etc are following the race. Why mobile applications?1. Have we ever thought that we would be connecting palm-sized projectors to our smartphones? First came the smartphones, then came apps for smartphones. Now, the next wave of innova- tion has begun, with the introduction of smartphone accessories that are controlled by apps. This opens up a whole new accesso- ries market and new opportunities to enterprising engineers. Refer to the December 2009 report from Gartner. Mobile phone users will spend $6.2 billion downloading applications in the year 2010, up from $4.2 billion last year. And the trend appears to only be starting with Gartner forecasting sales in mobile application stores of about $29.5 billion by the end of 2013. These are just the paid applications. The vast majority of downloads, an estimated 82 percent, are free to consumers. Mobile Application Stores’ Number and Revenue, Worldwide 2009 2010 2013 Downloads (in M) 2,516 4,507 21,646 Total revenue (in $M) 4,237.80 6,770.40 29,479.30 Table 1 (Source: Gartner, December 2009) Context for Mobile applications2. For mobile applications, the context is very important. Supposing, a sentence is typed in the text editor, then the editor will, after noticing a full stop, make the next character in the upper case. Similarly, if a driver is driving in a car on the highway, then the mobile device should, without much disruption, be able to pro- vide the meaningful content precisely and quickly. This is where the mobile devices are different to their desktop counterparts. Merely copying or porting the application from the desktop into the mobile domain will not serve the purpose of the user. The context for each mobile user can be different. The user pays for the content while being on the move, for each GPRS packet that gets transacted. For the business user, time is critical, and he can- not afford to spend time to retrieve desired details by browsing pages after pages like with a desktop. Similarly, if the movie op- tion or pizza outlet is to be explored in your local city, then there is no point in having details of movies in some other city. With GPS, the mobile device can understand the present location, which can help in providing movie details within the local city. Mobile apps classification3. The mobile applications can be classified broadly as: Platform applications• Web applications• Mobile websites• SMS• Platform applications3.1. The platform or native applications are developed specifically for the target CPU that the mobile device contains.The common plat- forms are Brew, Palm, Blackberry, .NET, Java ME, Linux, Android,
- 3. 91The Magazine for Professional Testerswww.testingexperience.com Mac OS, and Symbian etc. The development of platform applica- tions is relatively simple. However, platform application develop- ment, testing & supporting of multiple-device platforms is ex- tremely costly. This limits the product scalability that can impact negatively on return on investment. Web applications3.2. The mobile web application is the most acceptable solution to the above problem of development & deployment for providing multi-device support. Mobile web browser is an application that renders content on the mobile device. It is independent of device, platform and OS. The web browser knows its limitations which enables content to adapt across multiple screen sizes, including orientation either portrait or landscape that is detected by the accelerometer. Mobile websites3.3. Due to the mobility context, it is critical to have simple and ac- curate information rather than an overload of tons of info, which has a bearing on time & cost.The acceptable mobile site segments are email, news, weather, sports, city guides, and social networks. The user seeks the information relevant to meet the individual needs depending on the task and location. In this regard, the per- sonal context is different from the desktop web. Mobile web access today suffers from interoperability and us- ability problems. Interoperability issues stem from the device fragmentation of mobile devices, mobile operating systems, and browsers. Usability problems are centered around the small phys- ical size of the mobile form factor, limited resolution screens and user input operating limitations. Mobile websites are required to be optimized for the content based on the constraints on mobile devices which are listed be- low. Lack of Javascript and cookies• – Excluding smartphones, most devices do not support client-side scripting and stor- age of cookies, which are now widely used in most websites for enhancing user experience, facilitating the validation of data entered by the page visitor, etc. Small screen size• – This makes it difficult or impossible to see text and graphics dependent on the standard size of a desk- top computer screen. Lack of windows• – On a desktop computer, the ability to open more than one window at a time allows for multi-tasking and for easy reversion to a previous page. On the mobile web, only one page can be displayed at a time, and pages can only be viewed in the sequence they were originally accessed. Navigation• – Most mobile devices do not use a mouse-like pointer, but rather simply an up & down function for scroll- ing, thereby limiting the flexibility in navigation. Types of pages accessible• – Many sites that can be accessed on a desktop cannot be accessed on a mobile device. Many devices cannot access pages with a secured connection, Flash or other similar software, PDFs, or video sites, although this has recently been changing. Speed• – On most mobile devices, the speed of service is very slow, often slower than dial-up Internet access. Broken pages• – On many devices, a single page as viewed on a desktop is broken into segments, which are each treated as a separate page. Paired with the slow speed, navigation between these pages is slow. Compressed pages• – Many pages, in their conversion to mo- bile format, are squeezed into an order different from how they would customarily be viewed on a desktop computer. Cost• – The access and bandwidth charges levied by cellphone networks are much, much higher than those for fixed-line internet access. SMS3.4. The SMS is the most primitive service that is available to the users of the mobile device. In its simple form, the user sends a query in the form of short code, in order to receive instantaneously the desired information, such as train or flight times, stock prices, savings account balance, etc. The only limitation is that it is text- based, and there is a 160 characters limit on the message content. It can also be used to trigger some tasks or activities, such as dis- pensing tea or coffee from the vending machine. Challenges4. Some of the challenges have been discussed in the above para- graphs. The consolidated list is as follows: Service provider or operator control on cost• Device fragmentation• Compatibility issues to support different Operating Systems• (OS) & development platforms - Inconsistencies in user experience due to incompat-• ibility XHTML, CSS, JavaScript to view the same code across browsers Browsers are still evolving: Not every browser is able to sup-• port mobile web standards and industry best practices Service Provider or operator control on cost – Boon or bane? Service providers’ or operators’ specific role is to provide the wire- less service over the cellular network. They need to establish, operate, provide 24/7 support, maintain the network in order to provide reliable service to the end users. They need to interact & maintain their relationship with subscribers. They have to do provisioning, handling, billing of the services. Additionally, they are forced to provide packaged devices – meaning they need to provide support for mobile device + content + service. For this service providers go for a tie-up with mobile device man- ufacturers. This creates competition among the service providers with many plans and campaign offers. Ultimately, this results in putting pressure on service providers to meet unrealistic require- ments, such as having to support too many devices in the device family and compliance to relevant certification processes. This eventually has an undesirable effect on the pricing structure. In packaging the contents of mobile applications, they are expected to be provided almost free of charge. This leads to undesirable effects on development resulting into fear of being driven out of business. Typically, the devices are sold at much lower prices with up to 60% discount. Best practices, industry norms such as W3C, .mobi5. The World Wide Web Consortium (W3C) mobile web initiative was set up by the W3C to develop best practices and technologies relevant to the mobile web. The goal of the initiative is to make browsing the web from mobile devices more reliable and accessi- ble. The main aim is to evolve standards of data formats for Inter- net providers that are tailored to the specifications of particular mobile devices. The W3C has published guidelines (Best Practices, Best Practices Checker SoftwareTool) for mobile content and is ac- tively addressing the problem of device diversity by establishing a technology to support a repository of “Device Descriptions”. mobileOK5.1. W3C is also developing a validation scheme to assess the readi- ness of content for the mobile web, through its mobileOK scheme, which will help content developers to quickly determine if their content is web-ready. The W3C guidelines and mobileOK ap-
- 4. 93The Magazine for Professional Testerswww.testingexperience.com proach puts the emphasis on adaptation, which is now seen as the key process in achieving the ubiquitous web, when combined with a Device Description Repository. MobiWeb2.05.2. MobiWeb2.0 addresses the usability and interoperability issues that are holding back mobile web access today. It will focus on mobile Web 2.0 applications based on technologies such as Ajax that can significantly increase the usability of mobile Web appli- cations. MobiReady5.3. mTLD Mobile Top Level Domain, Ltd., usually known as dotMobi, is the Internet Corporation for Assigned Names and Numbers (ICANN), the appointed global registry for the .mobi domain. Backed by 13 leading mobile and Internet organizations, .mobi ad- dresses the need for seamless access to the Internet from mobile phones. DotMobi is the first and only top-level domain dedicated to us- ers who access the Internet through their mobile phones. With four mobile phones purchased for every one personal computer, there’s a world of people for whom the mobile phone is the main access point to the Internet. And those users can trust that a web- site is compatible with their mobile phone if the site’s address ends in .mobi. mTLD, the registry for .mobi, has released a free testing tool called the MobiReady Report to analyze the mobile readiness of web- sites. It does a free page analysis and gives a Mobi Ready score. This report tests the mobile-readiness of the site using industry best practices & standards. The single page test provides: dotMobi compliance• W3C mobileOK tests• Device emulators• Detailed error reports• Test strategy – Multivariate Testing (Multivariate6. Testing - Why, what and how?) Why - Multivariate Testing? It has been described that developing and deploying native plat- form applications for each device is an extremely costly affair. The simplest solution to the problem are web applications. Web applications can be independent of platform, OS and browser. It is relatively easier to support the device families. However, test- ing has to be conducted for every individual device in order to check the rendering of the application on the device, which varies in screen size. In view of this, the content that is being provided by the mobile website is the key. The context to the mobile user is extremely important and has a personal relevance, which has unique significance in providing the mobile application. The ren- dering and the amount of data and the associated time are very critical. This is going to be taxing to the end user if it has not been designed correctly. From this perspective, it is important that the website shall provide the information to the user based on his needs and expectations. You cannot afford to assume anything about the user. It is expected that the high-end device experience must be provided in the same manner to the lowest featured device in the family. The judgement of the developer and or the service provider is less important. The mobile is the most person- al mass media, unlike any other media such as radio, TV, movie, print media etc. It is to be noted that the mobile device contains almost all the previously established mass media incorporated into one single media. In view of this, it is proposed to optimize the mobile website using Multivariate Testing. The only way to figure out what content will work best on the mo- bile website is to test different content. Multivariate test strategy will help you study the effects of different content on your users. Testing will enable you to identify what users respond to best, so you can create a website that will be more effective in getting the business results you want. What - Multivariate Testing? A/B Testing Vs. Multivariate Testing A/B testing is used to test a few different versions of one single element on a page, e.g. title, font, background color, image, size of image, border thickness, text content etc. by creating variations in different page version. It will contain one reference page and one or multiple content variation pages. It is possible to conduct A/B tests with just two pages as a minimum requirement. A/B tests require a few hundred conversions taking place over a maxi- mum of several weeks for conclusive results in order to determine the winning combination. It is possible that at the end of the A/B test there may not be a conclusive result. If the results are inconclusive, this may mean that we weren’t able to sample a significant number of users, or there wasn’t a significant difference between the combinations tested. If there wasn’t a significant difference between combina- tions, but several combinations performed better than your origi- nal content, you may wish to replace the original content on your live site with the HTML content for one of the better-performing combinations. Alternatively, you may prefer to edit and run your experiment again. Thus it will be an iterative process to conclude based on the goals and objectives of the test. In this manner it will be an adaptive process that fits into Agile methodology. It means do not create big changes, even with the small changes in the iterative manner the site optimization experiment can be conducted for achieving the success of the website. Multivariate testing is a process by which more than one compo- nent of a website may be tested in a live environment. It can be thought of in simple terms as multiple A/B tests performed on one page at the same time. A/B tests are usually performed to de- termine the better of two content variations, multivariate testing can theoretically test the effectiveness of limitless combinations. The only limits on the number of combinations and the number of variables in a multivariate test are the amount of time it will take to get a statistically valid sample of visitors and computa- tional power. Multivariate testing is usually employed in order to ascertain which content or creative variation produces the best improve- ment in the defined goals of a website, whether that be user reg- istrations or successful completion of a checkout process. How - Multivariate Testing? There are two principal approaches used to achieve multivariate testing on websites. Page Tagging• By establishing a DNS-proxy or hosting within a website’s• own data center Page Tagging is a process where the website creator inserts JavaScript into the site to inject content variants and monitor visi- tor response. Page tagging usually needs to be done by a techni- cal team and typically cannot be accomplished by a web marketer or a business analyst. The second approach used does not require page tagging. By es- tablishing a DNS-proxy or hosting within a website’s own data center, it is possible to intercept and process all web traffic to and from the site undergoing testing, insert variants and monitor visi- tor response. In this case, all logic resides in the server rather than browser-side, and after initial DNS changes are made, no further technical involvement is required from the website point of view. Test setup description7. In this article for Multivariate Testing sample example is demon-
- 5. 94 The Magazine for Professional Testers www.testingexperience.com strated using Page Tagging. As already pointed out, Multivariate Testing is multiple A/B tests performed on one page at the same time. For example in this case the A/BTest is under consideration.There are many commercial outsourced test providers available. There are proprietary tools available or service is available from service providers. In the present case, an open- source tool from Google is used as an illustration for A/B Testing. The tool used is Google Web Site Optimizer (GWO). In our case the objective of the test was to check the tool usefulness and to demonstrate the mobile website optimization test result using GWO. Testing is carried out on a hosted mobile website by setting up the server to display the different variations of content, such as heading, different images, image sizes, background color, etc. to incoming visitors. Mobile website hosting There are in total three web pages for the .mobi website. The home page is the reference page. The alternate page is designed to create the variation in the reference page. The third page is the conversion page, which is just the reference conversion display- ing a thank you message. The reference to the hosted mobile website is provided as below. http://hebsyesh.mobi/hebs%20home.htm The tagged scripts are added to the web pages as per the GWO tool-generated JavaScript. The uploaded pages contain these tagged scripts. The web page testing is conducted using the mo- biReady online tool. http://ready.mobi/launch.jsp?locale=en_EN The test report summary is as below. It will probably display well on a mobile phone. Your mobi.readiness score is calculated from the results displayed below. Failing tests and large page sizes both lower the score. Read through the report to understand how to improve your score - and your site. Test summary 20 Passes 2 Warnings 4 Fails 2 Comments GWO Tool Report Statistics about total number of visitors to each page is measured by the GWO. It also indicates the total number of conversions. The report page is shown as an illustration. The report table indicates the original reference page visitors as 23 and conversions as 19. ForVariation1, the visitors are 21 and con- versions are 16. It still continues to collect the data in order to get a conclusive result. Testing challenges8. Some challenges were encountered: Hosting the .mobi web site• Testing the website with adequate user load• The GWO recognizes visitors with user visit if they are separated by distinct IP addresses. There are commercial and open-source performance testing tools available in order to create virtual user loads.The commercial tool access and availability is the challenge. Skill is required to do the needed code configuration. However, to create the user load in the order of 500 user,s the support for IP spoofing is required, which needs specific skills. If you visit once the web page for the configured website under GWO, it takes sometimes a few hours to count and reflect it in the GWO report as a visitor and converted page if applicable. Test results & analysis9. Refer to the mobiReady mobile website test tool test report. It has conducted testing dotMobi compliance• W3C mobileOK tests• Device emulators• Detailed error reports• There are 4 failures listed below. The detailed failure description is available in the online report. Your page does not use XHTML Mobile Profile• Your page markup does not validate• Incorrect or missing character encoding• No cache control headers were sent with this page• These can all be fixed based on the detailed description analysis provided in the online report. The GWO tool report is inconclusive at this stage, in order to de- cide on winner or loser. However, it has provided the conversion rate. The main reason is that no adequate number of visitors is available at this stage. It should be noted that this is due to re- source constraints. The testing continues till the sizeable number of visitors in the order of 500 is counted. Conclusion & recommendations10. Multivariate testing is currently an area of high growth in Inter- net marketing, as it helps website owners to ensure that they are getting the most from the visitors arriving at their site. Areas such
- 6. 95The Magazine for Professional Testerswww.testingexperience.com as search engine optimization and pay-per-click advertising bring visitors to a site and have been extensively used by many organizations, but multivari- ate testing allows Internet marketers to ensure that visitors are shown the right offers, content and layout to convert them to sale, registration or the desired action, once they have arrived at the website. Multivariate testing can also be very useful in other scenarios, for example to conduct opinion polls, free online services & applications survey, online print media, non-governmental organizations for understanding needs of targeted audiences within the society. Multivariate testing for mobile websites It is concluded that mobile and computing aspects can be effectively ad- dressed with optimized mobile website design as a part of user acceptance tests. In this context, the innovation is user-driven rather than by trying to sell the next big thing. Business analysts and marketing professionals have to play an important role when designing the test, understand the user con- text covering different geographic locations and optimize the tests. This is an iterative exercise. This really can be decisive for the value added to us- ers. It is also important to find out which portion of the web content is not accepted by the user, so that it can be avoided and lessen the time & cost burden. Ultimately, it is realized that multivariate testing is the most effec- tive scientific technique to design websites based on the user expectations. Mobile web analytics is an emerging area in the mobile net domain. It is strongly recommended to adopt multivariate testing at the user accep- tance level in the lifecycle model for mobile website optimization. This can be an ongoing exercise if true potential is to be achieved, and in order to stay competitive. It is better to know which mobile content offerings will con- vince and engage consumers most effectively, and then act upon the results. Iterative incremental changes can produce the best results, which is the best fit under Agile methodology. References: Mobile Design and Development by Brian Fling; Publisher: O’Reilly1. Gartner press release Dec 2009 http://www.gartner.com/it/page.2. jsp?id=1282413 “Gartner Says Consumers Will Spend $6.2 Billion in Mobile Application Stores in 2010” Mobile Web http://en.wikipedia.org/wiki/Mobile_Web3. Effective smartphone accessory design http://www.embedded.4. com/222400483?cid=NL_embedded Mobile Web Optimization Via Multivariate Testing http://www.media-5. post.com/publications/?fa=Articles.showArticle&art_aid=83319 mobiReady Online Tool http://ready.mobi/launch.jsp?locale=en_EN6. Google Website Optimizer Online Tool https://www.google.com/ac-7. counts/ServiceLogin?service=websiteoptimizer&continue=http:// www.google.com/analytics/siteopt/%3Fhl%3Den&hl=en Yeshwanta Hebalkar works at Mahin- dra Satyam’s Group / Satyam Comput- er Services Ltd., Hyderabad, India. Cur- rently, he is working as Consultant in various IT projects and programs. In the past, he worked with Invensys India Pvt Ltd., Danlaw Technologies India Ltd. His interests are to provide vertical-based solutions using automated combina- torial, multivariate testing and web analytics. He holds a Masters degree in Electronics as major from Fergusson College, affiliated to Pune University India. He started his career as senior research assistant in IIT Mumbai. He has experience of two decades in Tele- com & Radar, Defense, Automotive and Consumer Electronics industries. He was awarded fellowship in Electronics and subsequently worked as Scientist in Defense R & D Organization from 1988 to 1999. He is the recipient of an award from Scientific Advisor in recog- nition to the significant contributions to phased array radar. Biography Training by www.sela.co.il/en Israel, India, Singapore and Canada