20111006 synergie informatique-label-qualityof-experience_en
0 likes498 views
This document summarizes the results of performance benchmark tests on Microsoft Office 365, IBM LotusLive and Google Apps messaging solutions. It describes the methodology used, including testing access to each portal from various locations in France every 15 minutes over 71 days. It also describes scenarios for sending and receiving emails between each solution and external email addresses, tested every 30 minutes over 30 days. The results showed portal access availability of 99.980% for Office 365, 99.977% for LotusLive and 99.975% for Google Apps, and average access times of 1,541 seconds, 1,097 seconds and 3,552 seconds respectively.
20111006 synergie informatique-label-qualityof-experience_en
- 1. Quality of Experience Performance Benchmark on Messaging solutions Microsoft Office 365, IBM LotusLive and Google Apps Webinar October 6 th 2011 Alexis Blandin, Cloud & Social Business Leader | Synergie Informatique Arnaud Bécart, International Presales Engineer| ip-label David Wais, Business Development Leader| ip-label
- 2. Synergie Informatique Cloud Services Provider, France Social Business Specialist Expert in messaging and collaboration solutions integration IBM, Microsoft and Google partner
- 3. ip-label.newtest Around the world, ip-label.newtest solutions measure the quality of applications from the user’s point of view: business applications as well as web, voice and even TV over IP platforms. Over 1500 direct and indirect customers rely on ip-label.newtest to validate the end-to-end reliability of their websites and critical applications worldwide
- 4. Cloud goes Mainstream “ Cloud Computing Market Will Top $241 Billion in 2020 ” – Forrester Research “10% of Enterprise Email Will Be Based on a Cloud/SaaS Model in 2014” – Gartner Innovation = Public versus Enterprise Social Network Mobility, highly connected, convergence Increasing Data SaaS, a serious consideration
- 5. Messaging in the line of sight Cloud Usage Scenarios CIO Expected Advantages 1. Prototyping Commodities delegation 2. Test & Development Industrialisation Fast deployment 3. Messaging Savings, ROI 4. Collaboration Self-service, flexibility / PAYG Innovation (« beta ») 5. Web Applications Standardization, integration New challenges (mobility, …) 6. Backup, Archiving
- 6. The CIO Evolution Guarantees, disponibility (SLA) Warranty and « Quality » improvement Security, externalisation, reversibility Sales relationships (B2B) Contracting and legal management, (Patriot Act, CNIL) Going from CAPEX to OPEX (subscription) Operational and organisational restrains
- 7. Adapted Needs Governance, « Cloud » Best Practices Adapted procedures flowchart More straight to the point (support, specifics, infrastructures) Dependant to the service/ network Metrology Tools User experience
- 8. Editor’s Focus / Collaborative Suite 1911 1975 1998 Identity, Services / Softwares Windows / Office Search Strategy Social Business « All In » Pure Player Product LotusLive Office 365 Google Apps For Business Acquisition By Yes Progressive (options) No modules €/U/Y 34 to 113 € 43 to 306 € 40 € (public prices) • Social and Collaborative • Pack Office Integrated • Web Experience Advantages Network • Sharepoint Framework • Google ecosystem • External Opening • Some missing features • SharePoint Out-of-the-Box • No web conference Weaknesses (online office suite, (ergonomy, features) • Very basic integrated social and blogs/wikis, call audio/video) • Document-Centric collaborative tools
- 9. E-Mail Systems Focus “ Inbox = Tetris getting zero lines doesn’t win it just makes the next move easy “ Product • LotusLive iNotes/Notes • Exchange Online • GMail • Codes different than a classic • Classic codes • Classic codes messaging suite Features • Native Lotus Notes included • Native Outlook integrated • GMail offline (Google Chrome) • Two offers: basic(web mail) + Technical advanced (Domino) • Hybride Mode • Full Web • Hybrid Mode • Leader On-Premise • Lack of visibility, can look strong • In position of conquest • Late Coming-Out Cloud Analysis, enough just for Domino clients • Big efforts made in some • Difficult move from BPOS to trend • Domino/LotusLive future vision points (sécurity / Office 365 : which available administration) consequeces ? Space 25 Go 25 Go 25 Go SLA, Credit according 99,9 % 99,9 % 99,9 % to the available 2 à 10% monthly cost 25 à 100% monthly cost 3 to 15 days %
- 10. Methodology Data Solution Pilot for all web applications , rich web or client ( Office 365, Outlook…) run like a user: mouse click, filling application forms, text control, click on image etc. Fast launch on one of the 70 measuring points in the world (5 operators are tested in France, Orange, Neuf-SFR, Iliad-Free, Colt, Cogent) Mode « Saas » on demand, from set up tests to operational robot and servers maintenance Access to all measures via an extranet, reports sent via mail and warnings on incidents Access to back up for all set up , expertise and training requests Scenari recording tools for clients wanted to be autonomous
- 12. Tests Description 3 platforms tested 3 types of tests – Access to the portal from 5 spots in France • https://portal.microsoftonline.com • https://apps.lotuslive.com • https://www.google.com/a/synergie-informatique.fr/ • Tests origins: France Telecom, Neuf-SFR, Iliad-Free, Colt, Cogent • Tests frequency: 15 minutes • Tests duration: 21/07/11 au 30/09/11 (71 jours) – Scenario SENDING from 1 spot in France • Sending a mail from the Cloud platform to a external messging address • Delivery mail authentication in less than 60 seconds • Tests frequency: 30 minutes • Tests duration: 1/09/11 au 30/09/11 (30 jours) – Scenario RECEIVING from 1 spot inFrance • Sending a mail from an external messaging address rto the Cloud platform • Receiving mail authentication in less than 60 seconds • Tests frequency: 30 minutes • Tests duration: 1/09/11 au 30/09/11 (30 jours)
- 13. Tests Description Scenario SENDING – Connexion to the portal – Authentication – Access to the e-mail – New online message to mailXX@ip-label.net – Send the message and wait for POP from ip-label (mail.ip-label.com) – waiting time max 60 seconds Scénario RECEIVING – Connexion to the portal – Authentication – Access to the e-mail – Send a mail via SMTP from a server from ip-label (mail.ip-label.com) – Wait for mail receipt on the tested platform – waiting time max 60 seconds Each sent mail is automatically deleted after each test Each portal account uses a distinctive external messaging account
- 14. Tests Description When the portal we are trying to access is considered unavailable? – Non loading of the portal in less than 60 seconds – Network incidents detected ( DNS faulty resolution , failed connexion) and server(loading slowness) When a scenario is considered unavailable ? – Non complete loading of a page in less than 30 seconds linked to network or server incidents – Keywords not found • Username to validate the loading of the portal • Username/Password, connexion button, then button ‘new mail to validate the authentication step • SEND : button new mail and send • RECEIVING : refresh button to verify the mail had been received – Non transmission of the e-mail ( external SMTP online messaging or online messaging SMTP) in less than 60 seconds
- 15. Portal Access Availibility and average access time to each portal Results from 21/07/11 to 30/09/11 – Around 34000 tests per portal 1,541 seconds 1,097 seconds 3,552 seconds 99.980 % 99.977 % 99.968 % google apps Résolution DNS lotus live Temps de connexion Temps de réponse du serveur Office 365 Chargement du contenu 0 500 1000 1500 2000 2500 3000 3500 4000 – Google has the best portal loading time – Lotus Live is quite fast (1,5 seconds) but the page is longer to start (600ms), with longer DNS resolution, connexion and server time – Office 365 is the longest to load it is also the heaviest (1 Mo, versus 17 ko for google and 32 ko for Lotus Live) – Lotus Live shows the best availibility with 7 failures, Google (8 échecs) and Office 365 (11 failures)
- 16. Portal Access Access to portal availibility Access to portal performance Evolution over a 3 months period 4000 100 100 99,99 99,98 99,98 99,98 99,98 99,98 3500 3750 99,95 99,96 3474 3432 3000 99,9 2500 1982 Office 365 2000 99,85 1357 lotus live 1500 1286 99,83 google apps 99,8 1000 1119 1113 1061 500 99,75 0 Juillet Aout Septembre 99,7 Juillet Aout Septembre Conclusion - For the LotusLive portal , the answering time network and server has increased since 31/08/11 ( for an identical content) - Main errors encountered - Google : 5 DNS resolution errors and 3 time outs Lotus Live : 4 interrupted connections and 2 unreachable servers Office 365 : 4 DNS resolution errors, 5 unreachable servers and 2 timeouts - A maintenance intervention on the Office 365 portal on 9/9/11 from 5am to 8;15am was not taken into account
- 17. Authentication Authentication availibility and online messaging access Results from 01/09/11 to 30/09/11 –About 2800 tests per portal Performance Authentication and online 99,76 messaging access Disponibilité 99,86 24h/24 99,17 11438 Disponibilité 99,89 Office 365 Heures Ouvrées 99,89 google apps 8h-23h 99,56 15238 lotus Live Disponibilité 99,54 19054 HNO 99,81 23h-8h 98,52 0 5000 10000 15000 20000 97,50 98,00 98,50 99,00 99,50 100,00 – Google is the strongest plaftform in terms of authentication 99.86 % in 24/24 – Office 365 99.76% in 24/24 on authentication (mainly linked to some timeouts and DNS errors); those incidents usually occur at night and from 8h to 23h the availibility is the same as Google – LotusLive has the most errors at the authentication (99.17%); even if those errors occur mainly at night time, (99.56% availibility between 8h and 23h), one incident last September 18th between 8h and 10h penalise their position on this scale – Google is the fastest platform to authentify yourself and access to messaging : 11,5 seconds, LotusLive 15 seconds ( in one page load in opposition to the others) – Office 365 is the slowest to reach online Outlook : 19 seconds
- 18. Authentication Authentication Incident for Lotus Live on 18/09 Timeout : incomplete page after login (88k out of 212ko expected) Page error after login l 18
- 19. Authentication Authentification errors examples Authentification timeout on Office 365 Authentification timeout on Google Apps
- 20. Authentication Examples : Authentication errors on Google Apps portal Erreur du serveur (HTTP 500) sur l’objet Dashboard qui devrait rediriger sur le portail Erreur Cas normal Impact utilisateur : le champ login absent de la page et l’utilisateur ne peut pas s’authentifier
- 21. Sending/Receiving mail Performance Average length of time to send and receive e-mails 12271 Temps de réception 23213 18895 google apps lotus live 7558 Office 365 Temps d'envoi 10049 15095 0 5000 10000 15000 20000 25000 – On average, to send a mail from a Cloud platform is faster than receiving a mail from an external platform – Google is the platform which transmits mails the fastest, as well as sending to an external account (12,2 seconds on average) and receiving from an external account(7.5 seconds) – Office 365 is faster than Lotus Live for receiving mail, but slower than LotusLive to send mails
- 22. Sending/Receiving mail Stability Average availibility for sending and receiving mails 86,17 Disponibilité de réception (<60s) 99,18 71,66 google apps lotus live 85,17 Office 365 Disponibilité d'envoi (<60s) 97,62 74,81 0 10 20 30 40 50 60 70 80 90 100 – Lotus Live is the most available platform in terms of sending mails, if we consider as unavailable a length of time superior to 60 seconds ,to send a mail CloudExterne or ExterneCloud – Office 365 (~75%) and Google Apps (~85%) : This means even with a fast sending and receiving mail process (<15 secondes), engorgements regularly slow down the e-mail sending or receiving process – Sending and receiving mail process is more unstable for Google and Office 365
- 23. 2 1 3 Google Apps Gmail can be considered as a reference in terms of online Scores Google Office messaging access speed and you are guaranteed fast sending and Apps Lotus Live 365 receiving mail Portal performance 3 2 1 Portal availibility 2 3 1 IBM Lotus Live is a stable platform in terms of sending and receiving Authentication availability and online messaging access 3 2 1 mails but can be a disappointment for users because of access Authentifcation availability unavailibility and average access time (15 sec) (working hours) 3 1 3 Receiving Time 3 1 2 Microsoft Office 365 has a good messaging availibility (99.89% during Sending Time 3 2 1 Receiving Stability 2 3 1 working hours) but some progress need to be made on page loading Sending Stability 2 3 1 time ( particularly on authentication and access to Outlook) and on Total 21 17 11 sending stability
- 24. Conclusions Results Details Gmail Google – Best Portal Access Performance – Best authentication performance and availibity – Best sending time to an external e-mail address and receiving from an external e-mail address – Relative unstability in transfert time frames • 15 mails out of 100 are sent in more than 60 seconds • 14 mails out of 100 are received in more than 60 seconds IBM Lotus Live – Look out on the highest portal loading time since the beginning of September – Best portal access availibility – Best transfert stability 1 mails out of 100 are sent in more than 60 seconds 3 mails out of 100 are received in more than 60 seconds – Authentication performance and availibility is not good Microsoft Office 365 – Very good authentication disponibility during working hours. – Portal access performance and access to messaging to be improved (19 seconds) – Stabilité des durées de transmission des mails à améliorer • 25 mails out of 100 are sent in more than 60 seconds • 28 mails out of 100 are sent in more than 60 seconds
- 25. Conclusions Each editor commits himself to 99.9% availibility – « Microsoft provides a financially-backed 99.9% uptime guarantee » • http://www.microsoft.com/fr-fr/office365/what-is-office365.aspx – « LotusLive Notes delivers the capabilities your business users need to increase efficiency while reducing IT complexity and associated bottom line costs - all backed by a dedicated IBM team with 24x7 support and a 99.9% up-time service level objective » • https://www.lotuslive.com/styles/tours/transcripts/LLNotes_datasheet.pdf – « Google Apps includes a 99.9% uptime SLA and synchronous replication » • http://www.google.com/apps/intl/en/business/messaging.html As a reminder data for September 2011… According to you, what is the quality really perceived by the online messaging users ?
- 26. Q/A Were mails send with attachments ? – No , only text content; the add on of attachments of different sizes could be a criteria to do more tests Which internet navigator was used for the tests? – Internet Explorer 8 Can those tests be run on a client (Lotus Notes, outlook) ? Yesthe robots from ip-label can simulate any user running windows Can this benchmark be testing with files /document sharing? – Sharing files tools can also be tested by the robots ip-label What about other navigators , especially if Microsoft Exchange is losing out on fonctionnality ? – The robots can test Explorer or Firefox; the multi navigators cans be given through other solutions like Cloud Observer for instance Can we audit audio / video flow? – Ip-label can measure those flows using different measuring solutions on a user standpoint (streaming, voice quality analysis, video tests /audio conference, etc.)
- 27. Thank You Synergie-informatique.fr @SynergieInfo