T-Mobile and Elastic
1 like4,722 views
This document summarizes presentations given by three T-Mobile employees on how they use the Elastic stack to support customer experiences. Calum Lawler discusses using Elasticsearch to analyze social messaging conversations and calculate metrics to optimize customer care response times. Michael Mitchell explains how they use Raspberry Pis and the Elastic stack for remote device testing. Jon Soini talks about moving beyond dashboards to dynamic Canvas visualizations for sharing Elastic insights.
T-Mobile and Elastic
- 1. Observing Personal Customer Experiences at T-Mobile Calum Lawler, Michael Mitchell, Jon Soini
- 2. Come work with us! t-mobile.com/careers @tmobilecareers Grab one of us after the talk, and let’s chat about the cool stuff you’re doing with Elastic...
- 3. ! Hello! We’re Jon, Calum, and Mike from the Product & Technology group. How we care for our chat customers, our app customers, and our internal customers using the Elastic stack: " Customer Care via Text & Social (Calum) " Remote Device App Testing with Raspberry Pis (Mike) " Sharing Beyond ‘Dashboards’... or Canvas 201 (Jon) 👉
- 5. Customer Care via Text & Social
- 6. Calum Lawler Sr Engineer, Software " 4 years at T-Mobile " ~3 years using Elastic " Background in embedded systems " Lead Engineer for two SMPD teams (Social Messaging Platform Development) @semtake
- 7. Social Messaging Allows customers to interact with Care asynchronously ○ Almost a million messages per day Multiple channels supported: ○ T-Mobile app on phone, chat on t- mobile.com, Twitter (@tmobile), Facebook (Tmobile), Apple Business Chat Almost all issues can be solved over chat ○ Bill issues, new lines, plan changes ○ more...
- 8. “Social Messaging needs Monitoring…” “If you could go ahead and make dashboards and alerts and alarms and predictions, that'd be great...”
- 9. What do we Care about? Time To First Response " Time from message arrival to first (non-bot) response - shorter is better Time In Queue " Time the message was in the queue, waiting to be assigned to an Agent - shorter is better Assignment to Response " Time from assignment to an Agent to their response - shorter is better etc…
- 10. Metrics, Conversations... Need to calculate my own metrics from the conversations… 1. Get conversations from REST API 2. Compute a bunch of metrics 3. POST directly into Elasticsearch 4. Rinse, repeat Sounds like a Lambda...
- 11. Do it with Python! Use the Elasticsearch Client: from elasticsearch import Elasticsearch - Use a Template in Kibana - Create a doc with the data - Monthly indices - POST! # Send data to elastic... # Create the index (if it doesn't exist)... now = datetime.now(pytz.utc) indexStr = INDEX + str(now.year) + "_" + str(now.month) try: es.indices.create(index=indexStr) except Exception as err: logger.info(err) # POST the data... try: res = es.index(index=indexStr, doc_type='group_msg_int', body=doc3) except Exception as err: logger.error(err)
- 12. Why Elastic? Speed!!! We use TTFR metric to route conversations to appropriate TEX center Routing service needs to know TTFR for a center quickly in order to route to unloaded centers Elasticsearch response times (<100 ms) make this possible
- 13. Dashboards!
- 14. Dashboards!
- 16. Machine Learning! Out of expected range - send an Alert to Slack!
- 17. Future " Watch conversations to enable real-time sentiment analysis " Build amazing visualizations in Canvas " More application performance monitoring for SMPD services
- 18. Remote Device Testing with Raspberry Pis
- 19. Michael Mitchell Principal Software Engineer " 4 years at T-Mobile " 2 years using Elastic " Tech Lead for CX Lab (Customer Experience Lab) " Telemetry, Observability, IoT, Mad Science @michaeljmitc hel
- 20. About Customer Experience (CX) Lab " Telemetry, Monitoring, and Observability " Web properties ○ t-mobile.com, my.t-mobile.com, metropcs.com " Android/iOS apps ○ T-Mobile app, T-Mobile Tuesdays, MetroPCS app " Customer care/retail experience support ○ iPad, Messaging/Social integrations " IoT & emerging digital ○ T|Vision, Home Internet, SyncUp Drive/Pets
- 21. Problem: Real Device Testing is Necessary Emulators limited (for us, especially), need support for: " Functional, performance, & availability " Android & iOS operating systems " Native & web experiences " Authentic network, SIM support, etc. " Support for all major testing frameworks
- 22. Problem: Remote Device Access is Hard Partners around the world need manual and automated device access: ○ Real hardware devices ○ On the T-Mobile network ○ Dark/unreleased devices Logistics are complicated ○ Shipping coordination ○ Device firmware updates ○ Microcells don’t represent T-Mo network ○ Security/legal risks for unreleased hardware
- 23. Real Problem: Phones to Cloud " Piles of phones to test " No USB cables long enough Photo by Samuel Zeller on Unsplash
- 24. Early Prototypes and POCs " Vendor solutions lacking ○ Limited cell/SIM support, delay on new devices, performance, cost, etc. " Open-source awesome ○ Open Smartphone Test Farm (OpenSTF) ○ Appium/Selenium Grid " Hardware options ○ Off-the-self servers, 4x4 Intel NUCs ○ Early on-prem POCs OK, cloud was harder ○ Problems with USB hubs, endpoint saturation
- 25. What about ARM?
- 28. Production Raspberry Pi! " Golang on ARM is great " Elastic Beats on ARM are great " Containers on ARM work great (beware arch-mismatch) " Stable automation platform ○ Graceful degradation, rolling reboots ○ Super-fast provision, flash SD card ○ Exactly 1 hardware failure (self-inflicted 12V magenta LEDs)
- 29. CX Lab Device Matrix: Unified Device Automation
- 30. Real-time Remote Access & Logging
- 31. Device Matrix Monitoring with Elastic Stack Raspberry Pis
- 32. Problem: Dynamic Content is Hard " T-Mobile app provides very custom user experience " More dynamic UX == harder testing " Enterprise complexity: client/server/content/ platform teams ○ Microservices, routing, DNS, certs, firewalls, IDS, zero- rating " Account & environment challenges " Changes happen all the time, CI/CD!
- 37. Machine Learning: Auto Detection & Alerting
- 38. T-Mobile CX Lab + Elastic Stack " Elastic stack critical part of success ○ Service/feature offering and ops perspectives " Key enablement: ○ Devs and QA can access/filter/alert device logs in real time ○ Metrics, monitoring, alerting for service performance & availability ○ Self-health monitoring of lab devices, providers, & backend ○ Dynamic, targeted visualizations with Canvas
- 39. Sharing your Elastic insights & moving beyond ‘Dashboards’... or Canvas 201
- 40. Hi, I’m Jon Soini Principal Technology Product Manager " ~ 4 yrs using Elastic stack " Worked w/ frontline customer care apps, the T-Mobile app, IoT hardware projects " ❤ ’s great ux, elk, data viz, R&D, search, mobile/IoT compute, design anything " Could give the talk: How not to run an Elastic cluster @soini
- 41. So...you’ve got data! It’s in Elastic! Hooray! But... Your target audience for your viz doesn’t want to actually use a dashboard OR You’ve got to get your insights to a passive monitor (lobby, operations wall, digital signage…) where it needs to look good
- 42. Let’s look at Canvas Where traditional viz’s are canned & configurable, Canvas starts blank, and lets you meld it to your liking. ...daunting Photo by Fabrizio Conti on Unsplash
- 43. Starting w/ a gauge Take the existing gauge widget… a few issues: " Smaller scales omit the value! " Number formatting limited " Colors are hard-coded " Metric itself not color coded " Hard to change on the fly
- 44. A simple gauge in Canvas - SQL filters | essql query=" SELECT AVG(launch_duration) as duration_s FROM "tmoapp*" WHERE launch_result='pass' AND device_platform='Android' AND timestamp > NOW() - INTERVAL 60 MINUTES"
- 45. A simple gauge in Canvas - formatting | math "duration_s" | progress shape="semicircle" label={formatnumber "0,0.00"} font={font family="Tele-GroteskUlt, 'Open Sans', sans-serif" size=36 align="center" ... weight="normal" underline=false italic=false} max=10
- 46. A simple gauge in Canvas - switch statement valueColor= {switch case={case if={all {gt 2} {lte 6}} then="yellow"} case={case if={gte 6} then="red"} default="green"} barColor="#BBBBBB" | render containerStyle={containerStyle}
- 47. Drawing the rest of the owl
- 48. Canvas 201 - Time Date | math "value - 25200000" | formatdate "h:mm a" | markdown "" {context} font={font family="Tele-GroteskUlt, 'Open Sans', sans-serif" size=20 align="left" color="#000000" underline=false italic=false} | render
- 49. Canvas 201 - Histograms SELECT HISTOGRAM(timestamp, INTERVAL 60 SECONDS) AS histogram, AVG(duration)/1000000 AS duration FROM "tmoapp_analytics-*" WHERE event_type='card_load' AND endpoint='{someurl.com}/card/group/ Account' AND timestamp > NOW() - INTERVAL 180 MINUTES AND timestamp <= NOW() GROUP BY histogram LIMIT 180
- 50. Canvas 201 - Imagery It’s a true canvas with Z- ordering, support for transparent PNGs, animated GIFs, go wild! Also – images can be loaded from any URL/ CDN, use to pull in dynamic imagery!
- 51. Canvas 201 - Text Inserts Filters | essql query="SELECT summary.up, count(distinct url.domain) as count FROM "heartbeat*" where "@timestamp" > now() - interval 15 minute group by summary.up" | markdown "Yo dawg, " {getCell "count" row=1} " services are online, and sadly ” {getCell "count" row=0} " are offline.”
- 52. Canvas 201 - Conditionals filters | <<some data source>>| head | getCell "my_apps_status" | if {compare eq to="healthy"} then="healthy-asset-guid" else={ if {compare eq to="unhealthy"} then="unhealhty-asset-guid" else={ if {compare eq to="unknown"} then="idontknow-asset-guid" else="fallback-asset-guid" ... CONDITIONAL BANNER BASED ON SEARCH
- 53. Jon’s Canvas Tips " Work out your SQL queries in the Dev Console first! " Download your workpad JSON as you go, store in version control " Optimize your images ~1080p res " Build one element fully, then clone " Group/ungroup elements with G and U keys " Tons of great examples on the Elastic Blog " Alex Francoeur’s examples on GitHub https://github.com/alexfrancoeur/kibana_canvas_examples
- 54. A few example canvases
- 62. Live long and prosper. " Come talk to us at the AMA booth! " Or find anyone with a T-Mobile t-shirt! " Slides & notes: https://github.com/tmobile/ elastic-demos @soini @semtake @michaeljmitchel