5733 a deep dive into IBM Watson Foundation for CSP (WFC)
1 like3,533 views
This document provides an overview of the Watson Foundations for CSPs (WFC) architecture, which uses four analytics components - discovery, detection, decision and drive - to enable use cases across various telecommunications business capabilities. It describes how WFC integrates predictive modeling using SPSS, real-time analytics using InfoSphere Streams, and other components to power analytics applications for areas such as customer experience management, fraud detection, location-based services and more.
![Acknowledgements and Disclaimers
Availability. References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in
which IBM operates.
The workshops, sessions and materials have been prepared by IBM or the session speakers and reflect their own views. They are provided for
informational purposes only, and are neither intended to, nor shall have the effect of being, legal or other guidance or advice to any participant.
While efforts were made to verify the completeness and accuracy of the information contained in this presentation, it is provided AS-IS without
warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this
presentation or any other materials. Nothing contained in this presentation is intended to, nor shall have the effect of, creating any warranties or
representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use
of IBM software.
All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have
achieved. Actual environmental costs and performance characteristics may vary by customer. Nothing contained in these materials is intended
to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other
results.
© Copyright IBM Corporation 2014. All rights reserved.
— U.S. Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract
with IBM Corp.
— Please update paragraph below for the particular product or family brand trademarks you mention such as WebSphere, DB2,Maximo,
Clearcase, Lotus, etc
IBM, the IBM logo, ibm.com, [IBM Brand, if trademarked], and [IBM Product, if trademarked] are trademarks or registered trademarks of
International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are
marked on their first occurrence in this information with a trademark symbol (® or TM), these symbols indicate U.S. registered or common law
trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in
other countries. A current list of IBM trademarks is available on the Web at
• “Copyright and trademark information” at www.ibm.com/legal/copytrade.shtml
• If you have mentioned trademarks that are not from IBM, please update and add the following lines:[Insert any special 3rd party trademark
names/attributions here]
• Other company, product, or service names may be trademarks or service marks of others.
36](https://image.slidesharecdn.com/5733-adeepdiveintoibmwatsonfoundationforcspfinal-141030011757-conversion-gate01/85/5733-a-deep-dive-into-IBM-Watson-Foundation-for-CSP-WFC-36-320.jpg)
5733 a deep dive into IBM Watson Foundation for CSP (WFC)
- 1. 5733 - A Deep Dive into Watson Foundations for CSPs (WFC) Architecture Dr. Arvind Sathi asathi@us.ibm.com Richard Harken rharken@us.ibm.com Tommy Eunice teunice@us.ibm.com Mathews Thomas Matthews@us.ibm.com Wed 29/Oct, 04:30 PM - 05:45 PM © 2014 IBM Corporation
- 2. Content • Watson Foundation for CSPs (WFC) • Discovery and Predictive Modeling using SPSS • Detection and Real-time Analytics using InfoSphere Streams • Component Integration • Conclusions
- 3. Real-Time Analysis and Event Processing (RTAP) BSS Netezza / ISAS / … Real Time Analytical Processing Data Warehouse Network OSS (Landing Area) Reference Data (from EDW) Input Data Mediation Our Telco Evolu,on – Ini,ally Just Media,on and Complex Event Detec,on
- 4. Mediation Real-Time Analysis and Event Processing (RTAP) DWH, Analytics Foundation (ELT & In-Database Analytics) Real Time Analytical Processing Data Warehouse Network OSS (Landing Area) Reference Data (from EDW) BSS Input Data Real-Time Event Triggers PMML Model Deployent Marketing Campaign System Mobile Advertising Predictive Analytics (SPSS, etc.) Analytical Monetization Platforms Source Systems Our Telco Evolu,on – Adding on Marke,ng and Predic,ve Analy,cs
- 5. Deep Packet Inspection Real Time Analytical Processing Network Analytics Network Analytics Insights (Dashboards) EDW Real-Time Event Triggers Marketing Campaign System Mobile Advertising Analytical Monetization Platforms Source Systems Network OSS (Landing Area) Reference Data (from EDW) BSS Our Telco Evolu,on – Adding DPI and Network Analy,cs BI & Visualization Customer Experience Management (Data Model) Other BI Dashboards & Reports Analytics Foundation (ELT & In-Database Analytics) CRM Predictive Analytics TNF Real-Time Analysis and Event Processing (RTAP) Input Data Mediation SPSS-Streams Toolkit TNF
- 6. Deep Packet Inspection Geo-Spatial Analytics Cyber-security Analytics Web Analytics Real Time Analytical Processing Network Analytics Network Analytics Insights (Dashboards) EDW Real-Time Event Triggers Analytical Monetization Platforms Source Systems Looking Ahead-‐-‐ More Analy,cs , Cybersecurity, BigInsights Marketing Campaign System Mobile Advertising BI & Visualization Customer Experience Management (Data Model) Other BI Dashboards & Reports Analytics Foundation (ELT & In-Database Analytics) CRM Predictive Analytics TNF Real-Time Analysis and Event Processing (RTAP) Network OSS (Landing Area) Reference Data (from EDW) BSS Unstructured Data Input Data Mediation Hadoop Distributed File System (HDFS ) Text Analytics Machine Learning Large Scale Analytics Big Insights Web Log Analytics Sentiment Analytics Topic-based Influencers Social Media Applications TNF SPSS-Streams Toolkit
- 7. Drive WFC uses D4 for tight integration across four analytics components Apply the results of inves1ga1on to take ac1on by interac1ng with the subscribers in real-‐ 1me. Collect feedback from ac1on for future analysis. Discover Collect historical behavioral data, past acts, and success rates. Analyze historical data to formulate pa?erns and changes required to detect, and inves1gate steps Decide Gather data on targeted customers from a variety of sources over 1me to establish behavioral pa?erns and iden1fy how to respond to an emerging pa?ern. Detect Detect in real 1me if a transac1on, request, applica1on, document, etc. is required for targe1ng. Flag the selected dataset and ignore the rest. Detect observa,ons about a target Take ac,on in real ,me – when it ma8ers Find new targets by analyzing historical data Iden,fy pa8erns over ,me and ac,ons required Drive Detect Discover Decide Target Subscriber 7
- 8. Drive Interact with the customer to seek permission to use loca1on informa1on and send campaign, record interac1on and results. Smarter Campaigns using D4 Discover Collect historical behavioral data, past acts, and success rates. Analyze historical data to formulate pa?erns and changes required to detect, and inves1gate steps Decide Use background informa1on, past campaigns, privacy preferences, customer reac1on to past campaigns, purchase intent, preferences expressed in social media to design campaign. Detect Detect in real 1me if a transac1on relates to targeted subscribers. Iden1fy, align, score, and send for further processing (e.g., a targeted customer driving towards mall) Detect observa,ons about a target Take ac,on in real ,me – when it ma8ers Find new targets by analyzing historical data Iden,fy pa8erns over ,me and ac,ons required Drive Detect Discover Decide Target Subscriber 8
- 9. Drive Take appropriate ac1on to minimize losses due to fraud. Record ac1ons and results for future analysis. Fraud Analytics and Management using D4 Discover Collect historical fraudulent pa?erns. Analyze historical data to formulate pa?erns and changes required to detect, and inves1gate steps Decide Use background informa1on, past usage, loca1ons, subscrip1on, bill payments to find if the fraudulent transac1ons are associated with a subscrip1on. Seek more data as needed. Raise an alarm. Detect Detect in real 1me if a transac1on relates to a fraudulent subscrip1on (e.g., inconsistent geography or usage in consecu1ve transac1ons). Send alert for further Inves1ga1on. Detect observa,ons about a target Take ac,on in real ,me – when it ma8ers Find new targets by analyzing historical data Iden,fy pa8erns over ,me and ac,ons required Drive Detect Discover Decide Target Subscriber 9
- 10. WFC Use Cases Business Capability Use Case Name 1 0 Business Intelligence Centralized Business Intelligence EDW/BI Transformation Competitive Monitoring Customer Experience Management Effective Customer Care Apologize for Poor NW exp Enhance Consumer Billing Reports CX for Roamers Personalized Experience Best Video Experience Customer Care for VIP Quality of Experience for Apps Service Trouble Shooting at C-Center Cross-channel Optimization Channel Optimization Lead Management Sales and Support Integration Shopping Carts and Lists Management Insight Analytics Data usage Patterns Multi-Sim Behavior Online Behavior – Trending Behaviors Online Behavior – website Analysis Roamer Behavior Voice/SMS Apps impact on Traditional voice and sms services Tethering Behaviors Business Capability Use Case Name Data Monetization Online Market Analysis Enterprise M2M Proposition Family Online Protection Sports Analytics Effective Advertising Media Metrics Monetize market data 3rd Party Advertising Networks Device Management Device Analysis Device SW version Upgrade Notifications Device Migrations Traffic Analysis Voice and Data Performance Fault detection/notification/NFF Dynamic Pricing & Service Models Advanced Thetering on Shared Billing Accounts Proactive High Speed Access at Partnered Locations B-width onDemand at Strategic Hotspots Field Service Management Agenda Optimization & Resource Activation at Customer Premises Fraud Management Equipment Fraud Subscription Fraud Identity Theft VoIP Hacking Dealer Fraud Business Capability Use Case Name Smarter Campaigns Improve Conversion rates by Timing offers effectively Contract Renewal/Retention/Up/X-sell Intelligent Data Services Content Based Advertising Content Based third-party Advertising based on Location Content provider Advertising QoS Charging for Content Providers Location-based Services Contextual marketing Co-Presence Dynamic AdHoc group Geo-fence Mktg Propositions Massive Events Mobility patterns Hang out Occupancy/Traversal/Anomaly Tariff plan innovation / optimization App/Location/Time- based Plans Tariff Planning Plans and Add-on Impact on App Usage Network and Service Optimization Service Usage & Usage Location Customer Centric NW Monitoring 3G Locked Subscribers IB roaming performance analysis Intelligent network policies Capacity Planning Subscriber and Service Management Interactive multimedia ticketing Mobile Real Time Social Campagn New Business Models- Loyalty Self Care Multi-Device Family Accounts
- 11. Content • Watson Foundation for CSPs (WFC) • Discovery and Predictive Modeling using SPSS • Detection and Real-time Analytics using InfoSphere Streams • Component Integration • Conclusions
- 12. Introduction to Subscriber Dimensions from Mobility Analytics Usage Style l Heavy Voice l SMS Mostly l No Data Interests l From DPI l Webpage analytics l e.g. Golf, Betting Quality of Service l From xDR l Network l The Now Factory Demographics l Based on usage patterns l Websites l Buddies Lifestyle l Commuter l Homebody l Night Owl Preferences l OTT Messaging l Travel, Games l Handset prefs Preferred Locations l Hangouts l Home Work l Mode of Travel Best Buddies l Who calls who l Who hangs out with Who?
- 13. How to turn streaming noisy Telco Location data into meaningful location, then discover customer insights Stream data Call Detail Records SMS Voice GPS Tracking Reference Data Cell Tower Wifi AP Maps GIS, POI Special Service Numbers e.g bank, 1-800 Analyzable Location Event Data Who, when, where and what subscriberId: Timestamp: Position: latitude + longitude Precision: 0~2 km Direction: nullable Speed: nullable Activity : nullable Meaningful Location subscriberId: home: Work: POIs & period … Sequence of meaningful Locations… Commute means: car/ subway/bus Micro segmentaton Business traveler Regular commuter Heavy driver Social Butterfly Mom ….. Location Patterns on Individual and Group level ü Every Sunday noon, Bob goes to xxx mall to shopping and has lunch ü Every Thursday afternoon, Bob goes to customer site at XXX ü ….. Mobile Location Data Processing: Map mapping, Business rules et. Big Data Integration Spatio-Temporal Event Association Analysis Wifi off load Location Pattern Analytics
- 14. Discovery using structured data • A typical discovery uses statistical tools to identify pattern in data. • Discovery may contribute new derived attributes for further analysis or reporting. Night Owls at Night Delivery People During the Day Quiet Weekday people go for dinner on weekends Almost no Homebodies any time
- 15. Mobility Lifestyles (developed by IBM) * from the Television show, “Cheers”. Norm was an accountant who went to the same pub every night
- 16. Mobility Lifestyles • How do the lifestyles of subscribers vary by location and time of day • Why do lifestyles matter for Retailers? § Certain lifestyles tend follow habits much more than others: Daily Grinders and Homebodies go to the same locations often and are predictable. The other lifestyles tend to be less predictable
- 17. Mobility and Usage Lifestyles Very distinct patterns, this level of differential is high
- 19. Location Analytics - Dashboard Screen shots from Cognos
- 20. Predictive Modeling and Scoring example
- 21. Content • Watson Foundation for CSPs (WFC) • Discovery and Predictive Modeling using SPSS • Detection and Real-time Analytics using InfoSphere Streams • Component Integration • Conclusions
- 22. Streams Data Processing in Telco Environment Decoding, filtering, aggregation, correlation, summation, transformation, formatting, ... Streams Telco Realtime Processing Performance Data CDRs Logs Event Data Configuration Data Telco Network Elements Source data format: ASN.1, XML, ASCII, binary Standardized or proprietary, via edge adapters Output into dashboards, databases, files Statistics, monitoring, archiving Tap into Message Transfer Telco Solutions
- 23. Visual Representation A New Paradigm: In-Motion analytics for High throughput and Ultra-low latencies Continuous Ingestion Continuous Queries / Analytics of data in motion Streams Application Data Sink Data Tuple Operator Data Sources InfoSphere Streams Overview
- 24. From Vast Data to Actionable Insights InfoSphere Streams addresses the challenge for CSPs is to turn the vast amounts of customer data they collect into usable and actionable insight Millions of events per second CDRs Billing CRM Location Account Mgt Internet Network Dropped Calls Outgoing Interna,onal Calls Call Dura,on Extra Call Invoice Issued Invoice Paid Contract Expira,on Acquired new products Change contracts Entered new cell Customer is roaming Customer is at home New Top-‐Up 5 minutes leM on pre-‐paid Changed Home Loca,on Brand Reputa,on Customer Sen,ment from Social network Broadband Satura,on Congested Cells Streams of Intelligence 3 dropped calls in 10 minutes Customer is close to a store Customer enters a shopping area Invoice paid + ‘liked’ compe,tor Smart phone browsing pa8ern Customer is watching a video Microsecond Latency Required Ac,onable Insight Who is THIS customer and what does S/HE want? MDM, EDW
- 25. Possible Architecture (for live system) Network/2 Internet Analytics 1 Data Collection TAP IBM CONFIDENTIAL Filter (Brocade, Gigamon, etc) Load Balance (Brocade, Gigamon, etc) ISP Blade 1 Blade 2 Blade 3 Blade 4 Blade 5 Blade 6 Blade 7 Blade 8 Blade 9 Blade 10 Blade … Each line represents a link to a physical network interface on a blade. It carries data for one or more pre-specified protocols. All packets belonging to one session are sent on one link. Netezza
- 26. Integration with Discovery 1. Data Collection and Pattern identification. 2. The offline modeling step- using the SPSS Modeler- creates analytic models based on labeled training data. The data can be hosted on any platform example: data warehouse, Pure Data Systems for Analytics, Hadoop. 3. The intermediate integration step - There are two alternatives to deploy SPSS models in streams. One is to generate the PMML model. There is a limited set of models that generate the PMML format. The PMML model is then deployed in the Streams mining toolkit. The other approach is to publish the model to generate the .pim, .par and .xml files which are supported by the SPSS Analtyics Toolkit for Streams. These files are then configured on the SPSS Modeler Solution Publisher. 4. The on-line phase– Using Sreams SPL (Streams Processing Language), streams developer further uses appropriate operators and input output definitions of the models. This enables realtime analytics. 5. The action is triggered for instance executing a mobile campaign when a defined threshold is interpreted in real time.
- 27. InfoSphere Streams Publish the Model SPSS Scoring node NETEZZA Published .pim, .par, .xml files generated from SPSS IBM SPSS Modeler Solution Publisher In-‐database mining During the offline phase, SPSS Modeler accesses the training data residing in the Pure Data System for Analytics and creates the Model Nugget. The Modeler ODBC node can access the database table’s definitions as well as the data, and retrieves the relevant training data according to the selection criteria. Once trained, the Modeler creates a Model Nugget which can be published to Streams.
- 28. DB2 InfoSphere Streams UNICA Campaign polling Worklight Server Mobile App IBM SPSS Modeler Solution Publisher Model execution with Streaming data
- 29. Content • Watson Foundation for CSPs (WFC) • Discovery and Predictive Modeling using SPSS • Detection and Real-time Analytics using InfoSphere Streams • Component Integration • Conclusions
- 30. WFC Application Architecture A B C D G AAP Capabilities High Performance Historical analysis Model Based Predictive Analytics Real-time scoring, classification, detection and action Visualize, explore, investigate, search and report High Performance Unstructured Data analysis Discovery Analytics Take action on analytics F Simulation Dashboards Information Interaction Streaming Data Categorize, Count, Focus Score, Decide Streaming Engine Analytics Engine Prediction / Policy Engine Sense, Identify, Align Reports Geo/ Semantic Mapping Outcome Optimization Model Creation Semi Structured Data Data Repositories Network Events Network Policies Continuous Feed Sources XDR Batch Data Data for Historical Analysis Deploy Model Historical Data Models In Database Mining Reports & Dashboards Ad-hoc Queries Campaign Mgmt. Pro-active Customer Experience Management Pro-active Network Mgmt Real time Scoring & Decision Mgmt. ... Actions Event Execution Policy Mgmt External Data Social 3rd party High Velocity High Volume Open API Customer Activities A C B D G Customer Care Customer Care Marketing Marketing Network Planning ... Network Planning ... NOC/SOC NOC/SOC Users Deploy Model Policy Management Standardize Deduplicate Identity Resolution Data Integration ETL Network Topology Data Application & Usage Data Customer Data Capture Changes Un- Structured Data Hadoop E E Structured Data Search, Pattern Matching, Quantitative, Qualitative F Insigh t EDW Advanced Analytics Platform Create & Deliver Smarter Services Transform Operations Build Smarter Networks Personalize Customer Engagements Database Server
- 31. A B C D G AAP Capabilities High Performance Historical analysis Model Based Predictive Analytics Real-time scoring, classification, detection and action Visualize, explore, investigate, search and report High Performance Unstructured Data analysis Discovery Analytics Take action on analytics F Database Server SPSS Simulation Dashboards Information Interaction Streaming Data Categorize, Count, Focus Score, Decide Streaming Engine Analytics Engine Prediction / Policy Engine Sense, Identify, Align Reports Geo/ Semantic Mapping Outcome Optimization Model Creation Semi Structured Data Data Repositories Network Events Network Policies Continuous Feed Sources XDR Batch Data Data for Historical Analysis Deploy Model Historical Data Models In Database Mining Reports & Dashboards Ad-hoc Queries Campaign Mgmt. Pro-active Customer Experience Management Pro-active Network Mgmt Real time Scoring & Decision Mgmt. ... Actions Event Execution Policy Mgmt External Data Social 3rd party High Velocity High Volume Open API Customer Activities A C B D G Customer Care Customer Care Marketing Marketing Network Planning ... Network Planning ... NOC/SOC NOC/SOC Users Deploy Model Policy Management Standardize Deduplicate Identity Resolution Data Integration ETL Network Topology Data Application & Usage Data Customer Data Capture Changes Un- Structured Data Hadoop E E Structured Data Search, Pattern Matching, Quantitative, Qualitative F Insigh t Enterprise Data Warehouse Advanced Analytics Platform Create & Deliver Smarter Services Transform Operations Build Smarter Networks Personalize Customer Engagements InfoSphere Streams SPSS ODM, Optim, Open Pages PDA Social Media Analytics Watson Explorer Cognos InfoSphere BigInsights IBM (Unica) Campaign ODM PDOA BPM TNF SourceWorks TNF Smart Works Watson Analytics WFC Application Architecture using IBM products InfoServer EA
- 32. Analytics Capabilities • Reporting § Structured, Unstructured, Ad hoc • Discovery § Structured, Unstructured • Predictive Modeling • Identity Resolution • Customer Profiling • Real-time Filtering § Static, Dynamic • Real-time Scoring • Simulation • Feedback and Machine Learning • Visualization 32
- 33. Content • Watson Foundation for CSPs (WFC) • Discovery and Predictive Modeling using SPSS • Detection and Real-time Analytics using InfoSphere Streams • Component Integration • Conclusions
- 34. Reading Material • IBM Developer Works § Explore the advanced analytics platform, Part 1: Support your business requirements using big data and advanced analytics § Explore the advanced analytics platform, Part 2: Explore use cases that cross multiple industries using the advanced analytics platform § Explore the advanced analytics platform, Part 3: Analyze unstructured text using patterns § Explore the advanced analytics platform, Part 4: Analyze location data to determine movement patterns using a mobility profile pattern § Explore the advanced analytics platform, Part 5: Deep dive into discovery and visualization § Explore the advanced analytics platform, Part 6: Dive into orchestration with a combination of SPSS, Operational Decision Management (ODM), and Streams using care and fraud management case studies • IBM Data Magazine § Mining Data in a High-Performance Sandbox - Fulfill data analysts’ dreams with data warehouse appliances for in-database analytics and data mining § Target Behavior in Real Time for Effective Outcomes: Part 1 - How real-time, adaptive architectures can drive management decisions for specific use cases § Target Behavior in Real Time for Effective Outcomes: Part 2 Drive marketing and business management decisions using a real-time, adaptive architecture • Books § Big Data Analytics: Disruptive Technologies for Changing the Game § Engaging Customers Using Big Data: How Marketing Analytics Are Transforming Business
- 35. We Value Your Feedback! • Don’t forget to submit your Insight session and speaker feedback! Your feedback is very important to us – we use it to continually improve the conference. • Access the Insight Conference Connect tool to quickly submit your surveys from your smartphone, laptop or conference kiosk. 35
- 36. Acknowledgements and Disclaimers Availability. References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. The workshops, sessions and materials have been prepared by IBM or the session speakers and reflect their own views. They are provided for informational purposes only, and are neither intended to, nor shall have the effect of being, legal or other guidance or advice to any participant. While efforts were made to verify the completeness and accuracy of the information contained in this presentation, it is provided AS-IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this presentation or any other materials. Nothing contained in this presentation is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results. © Copyright IBM Corporation 2014. All rights reserved. — U.S. Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. — Please update paragraph below for the particular product or family brand trademarks you mention such as WebSphere, DB2,Maximo, Clearcase, Lotus, etc IBM, the IBM logo, ibm.com, [IBM Brand, if trademarked], and [IBM Product, if trademarked] are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (® or TM), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at • “Copyright and trademark information” at www.ibm.com/legal/copytrade.shtml • If you have mentioned trademarks that are not from IBM, please update and add the following lines:[Insert any special 3rd party trademark names/attributions here] • Other company, product, or service names may be trademarks or service marks of others. 36
- 37. Thank You