Using Big Data and AI for Customer Analytics
- 1. Using Big Data for Customer Analytics at Transamerica David Beaudoin Vishal Bamba John LoGiudice Enterprise Computing Community Conference Marist College June 12-14, 2016
- 2. About Transamerica Transamerica is the US-based brand of Aegon, a Dutch financial services firm. Aegon is one of the world's leading providers of life insurance, pensions and asset management and is helping approximately 30 million customers globally to achieve a lifetime of financial security. 2
- 3. About Transamerica Operating under the Transamerica brand, the Americas is Aegon’s largest market, with two-thirds of the company’s underlying earnings generated here. Transamerica products and services help more than 27 million customers protect against financial risk, build financial security and create successful retirements. 3
- 4. About Transamerica Transamerica is among the top ten largest providers of variable annuities, individual universal life and individual term life in the US. In 2015, over $6B in benefits were paid. With over $100B in assets under administration, we serve more than 4 million retirement plan participants across the entire spectrum of defined benefit and defined contribution plans. 4
- 5. As you may have guessed… That is a WHOLE LOT of data to manage! 5
- 6. Data Architecture • Rich data environment across organizational business units, comprised of many source systems across various platforms… • A consistent enterprise view of data across business units is required. 6
- 7. Data Architecture 7 As in many industries, we are focused on leveraging technology to build data-driven customer relationships. How can the current data architecture support this strategic direction?
- 8. Data Architecture 8 Enterprise Data Management Crossroads! Another traditional warehouse project? OR Enterprise data hub/lake/ocean with new “Big Data” technologies?
- 9. A New Strategy for Customer Data… EMAP: Enterprise Marketing Analytics Platform • Integration of both internal and external data sources to provide a 360-degree view of customer relationships for marketing, planning and operational analytics. • Solution must provide strong data governance and security, and facilitate the use of clean, trustable, high quality data by business stakeholders and end users. 9
- 10. EMAP – Customer 360 What is your current & future value to the business?
- 11. Initial Use Cases Customer 360 Investment Advisor and Producer Profile Journey Map Customer Lifetime Value Marketing Attribution Asset Retention Foundational Applied
- 12. Enterprise Data Hub / Data Lake 12
- 13. Customer Data Lake – Iterative Approach
- 14. Big Data Management Data Integration HDFS Big Data management Data Quality Hive Hive / Impala MR / Spark Cleansed Files Individual Household Informatica Big Data Management Cloudera Big Data Platform Big Data Analytics Datameer Extract Load & Transform Data Quality –Cleaning, Identity Resolution Admin Extracts Partner Files IVR Enrichmen t Inputs CRM Solicitation History Weblogs BI Tableau H2O – ML Consumption Big Data Relationship Management Identity Resolution Search - Solr EMAP Solution Architecture
- 16. Identity Resolution Data Quality Ingest & Profile Cleanse & Transform Standardize / cleanse phone, email, address DOB BDM Tool: Developer & Analyst Tool Source 1 Source N Masking Cleanse, Standardize, Mask Personal Data BDM Tool: Developer BDRM matching/linking (assign cluster ID) Pre BDRM Managed Views Pre BDRM Managed Views BDM Managed Views (Current) BDM System Tables (Date) Hive BDRM Hbase/Hive Post BDRM Managed Views Post BDRM Managed Views Post BDRM Managed Views Profile Data Discovery Design/Define rules Build/Define Ref Table BDM Tool: Analyst Tool Address Doctor process EMAP Data Process Flow
- 17. Informatica Big Data Developer
- 18. HDFS Data Management Structure 18 Managed by IT • Core set of Informatica developers to create the mappings to ingest data from sources into HDFS (jobs run and monitored by IT) • Users can request access via a defined security approval process (Access can be limited at the file level) • Access will be tracked and can be included in auditing reports/events • SSN will only be allowed in the core data ingested from source systems. Managed by the Team • Governed onboarding process: IT will assist the team with setting up directories, reviewing recommended data flow process, reviewing security privileges, education on platform security and compliance issues, and providing basic setup guidance and assistance • Access will be tracked and can be included in auditing reports/events MANAGED DATA TEAM DATA Physically Separated by Directories
- 19. Audit Events Audit events/actions Include: • Failed login attempts • Role changes • Data access • Denied data access • Any many more … Alerts -- Can email an alert when a specific event occurs Data captured about the audit event: • Date • Command performed • Object affected • User that performed the action • IP Address Audit Events ARE NOT Stored on the Hadoop Cluster
- 20. Summary of Key Benefits • Provides a single platform to house key customer and prospect data sources • Establishes persistent keys across previously disparate data sources • Provides for rapid intake of new data sources (structured and unstructured) • Eliminates today’s data intake and append bottleneck • Empowers analysts to explore all data elements • Increases processing power for statistical analysis • Improved recruiting and retention of Data Engineers and Data Scientists 20
- 21. Lessons Learned • Tie to business use cases to demonstrate value • Align with larger enterprise strategy (business and technical) • Socialize the platform and vision • Technology is changing rapidly • Partner with key vendors • Partner with business • Invest in a PoC • Small team with the right skillset - innovative and curious 21
- 22. Lessons Learned • Big Data requires Data Governance • Establish tools and processes to support data governance from the start • Important to have Data Stewards: Profile, validate, catalog, metadata creation, lineage 22
- 24. Questions?