Flink Forward Berlin 2017 Keynote: Ferd Scheepers - Taking away customer friction through streaming analytics
6 likes1,514 views
ING is transforming into a data-driven organization by leveraging real-time analytics with technologies like Apache Flink and Kafka to enhance customer experiences and streamline operations. The bank's strategy focuses on understanding customer behavior, innovating beyond traditional banking services, and creating a modular bank architecture that promotes proactive customer interactions. This initiative is supported by an open metadata and governance platform to efficiently manage data across diverse systems and enhance decision-making capabilities.
Flink Forward Berlin 2017 Keynote: Ferd Scheepers - Taking away customer friction through streaming analytics
- 1. Taking away customer friction through streaming analytics FlinkForward Berlin Ferd Scheepers. Chief Information Architect ING. How ING uses data in real time with Apache Flink to enable it’s data driven journey Sept. 2017
- 2. Market leaders Benelux Growth markets Commercial Banking Challengers The world of ING- The best global bank in the world according to Global Finance magazine 2 Customers 37 Million Private, Corporate and Institutional Customers Countries more than 40 In Europe, Asia, Australia, North and South America Employees 52,000
- 3. 3 1. Earn the primary relationship 2. Develop analytics skills to understand our customers better 3. Increase the pace of innovation to serve changing customer needs 4. Think beyond traditional banking to develop new services and business models Empowering people to stay a step ahead in life and in business. Simplify & Streamline Operational Excellence Performance Culture Lending Capabilities Purpose Customer Promise Strategic Priorities Enablers Creating a differentiating customer experience Clear and Easy Anytime, Anywhere Empower Keep Getting Better
- 4. Trends in the banking landscape continue to evolve, our strategy is there to adapt and come out on top 4 Customer Behaviour Competitive Landscape Technology Fintech SocietyRegulation People’s time is precious; they don’t want to spend it on finance Products have become commoditised; the only way to differentiate is through the experience Regulatory uncertainty continues Digitalisation is erasing borders Ability to leverage new technologies will define future competitive advantage
- 5. “ING is an IT company with a banking license” – Ralph Hamers 5
- 6. In 2017, a few companies are making huge amounts of money with data. The data world is maturing. 6 The Economist: “The world’s most valuable resource is no longer oil, it’s data”
- 7. 5 years ago, everybody needed data scientists, now we need data engineers, we are out of experimentation only. 7
- 8. Our data driven journey started around 5 years ago, even before the CEO vision. Enter the ING Data Lake architecture. 8 Governance, Risk and Compliance Platform Events to evaluate Information Service Calls Data Load Data Feeds Information Service Calls Data Export Search Requests Report Queries Understand Information Sources Understand Information Sources Deploy Decision Models Deploy Real-time Decision Models Understand Compliance Report Compliance Information Service Calls Data Export Advertise Information Source Information Owner Information Systems System of Record Applications Distribution Layer Customer Centric Core Layer EnterpriseServiceBus New Sources Third Party Feeds Internal Sources Fulfilment Layer Generic & Support Services Layer Data Lake INFORMATION WAREHOUSE DEEP DATA Advanced Data Provisioning Catalog Interfaces Other Data Lakes Other Data Lakes Inter-lake Exchange Line of Business Interaction Data Marts Search Data Information Integration & Governance INFORMATION BROKER OPERATIONAL GOVERNANCE HUB CODE HUB Decision Model Management 10001 01011 01101 Line of Business Insight Simple, Ad Hoc Discovery and Analytics Reporting DataLakeRepositories Shared Operational Data ASSET HUB ACTIVITY HUB OPERATIONAL STATUS Descriptive Data INFORMATION VIEWS CATALOG Deposited Data Harvested Data DEEP DATA INFORMATION WAREHOUSE Data Lake Operations MONITOR WORKFLOW Self-service Data Access Deploy Real-time Decision Models STAGING AREAS Enterprise IT Interaction Information Ingestion Publishing Feeds Real-time Interfaces Generic Real-time Analytics Decisions STREAMING ANALYTICS Notification Events
- 9. In 2015, we started to work on the Touch Point Architecture, loosely based on platform thinking in the automotive industry. 9
- 10. We identified the essential components of the new modular bank, focussing on the customer and customer interaction 10 Authentication/Security provides the components and their interactions that facilitate a shared security architecture, covering Authentication, Security Tokens, Authorisation and Risk Engine interaction. Authentication/Security is based on the principles of ‘Stateless Design’, ‘Use of industry standards’, ‘Channel-agnostic Authentication means’ and ‘Means-agnostic Business API’s’. Notification brings the bank to its Customers. A push mechanism shifts ING towards becoming a proactive bank versus a reactive bank. We will provide meaningful information to our Customers at all time. We know the customer extremely well and add value by using pro-active notifications. Standardised Global Party, Product & Arrangement Management (PPAM) defines a common data exchange model and a set of interaction patterns that support globally reusable components with regards to parties, products, agreements and mandates. All PPAM processes, exchange models and data models are aligned in a standardised Global PPAM. Global PPAM has a standard interaction pattern and a single API, with no boundaries between segments and countries. Next to authentication, there is a specific Authorization workflow to request explicit consent of a customer (by ‘signing’) for executing actions like a payments or changing an agreement. Customer Context is about what we already know about the customer (previous locations, arrangements, his financial position, etc.), about the environmental context (world news, news about ING, what happens in my region) and what we know about the current customer’s touchpoint (location, device, time). Events are about a specific touchpoint, and as such influence the current interaction, but can also update person context as well as environment context for future analytics.
- 11. 11
- 12. Offering our customer actionable insights is the foundation of the new banking experience 1212 Relevant Actionable Real Time Personal
- 13. To deliver on the promise of TPA, we combined some workstreams, and started to look at solutions already in place. 13
- 14. Except for Coral, none of the existing solutions really suited the architecture, and Coral turned out to be hard to develop. 14 Streaming Computing Web, Apps and Services Event Bus Big Data (Not Realtime) other data sources, Lakes, SoR and services REST API STREAM API OLTP API Data Science Scenario testing ● Low Latency Data store ● Streaming computing on event Bus ● In-Memory Big Data computing Data Store Mobile and Web
- 15. As we started looking at patterns for streaming analytics, they always looked alike. Fraud and customer use cases are identical. 15 Producer Raw event What is it Relevant event What to do with it Outcome Producer Producer Producer CEP Filtering Enriching Rules Model scoring Artificial Intelligence Machine Learning
- 16. To re-use all the analytics, and to share state, we decided on one global platform for all streaming analytics in ING. 16 And we did a beauty contest for the technology choice
- 17. The result is an Event based architecture, with Apache Kafka and Apache Flink at the heart of it. Delivered as one platform. 17
- 18. CCN The Operating Model for all TPA platforms is global, one tribe, one product owner, a platform squad, potentially satellite squads 18 Platform tribe Flink Platform Squad SAS platform squad X Z Y Feature squad1 Feature squad 2 Satellite Satellite Satellite Organization • Platform team builds the core • Satellite teams across different regions do first line support • Feature teams build on top of the platform Code base • Maintained by Platform team Deployment • Handled by platform team • Multiple instances in different regions under same Life Cycle Management Use case jobs • Build and maintained by feature team Support/ Monitoring • Platform squad and satellite squads will both perform monitoring. • First-line support will be done by the satellite squads. Second-line support will be done by the platform squad.
- 19. Multiple startups have declared ETL dead, and that streaming will take over. Is batch just a slower version of real time? 19
- 20. But I have a different opinion on where we are in this journey 20 Ferd Scheepers, Chief Information Architect ING
- 21. Choosing how to address the needs of the enterprise will determine who will succeed in this space. 21
- 22. 22 ING is championing an Open Metadata and Governance that will allow… … metadata to be captured when the data is created, moved with the data and be augmented and processed by any of the vendor tools.
- 23. All enterprises have a heteregeneous data landscape, and the need for managing meta data across technology boundaries. 23
- 24. • 24 The Open Metadata and Governance initiative will allow easy implementation of meta data in any platform • Common base: • Automate the collection, management and use of metadata across an enterprise An enterprise catalog of data resources that are transparently assessed, governed and used in order to deliver maximum value to the enterprise. an open set of APIs an open set of metadata types an open set of exchange protocols Governance Access Frameworks Discovery
- 25. 25 We will deliver through Apache Atlas both a meta data highway, and an implementation that can be used by all Open and Unified Metadata Metadata repository Apache Atlas Metadata repository IBM Metadata repository Flink Open Metadata Repository Service OMRS Open Metadata Access Service OMAS Components defined and being developed by Open Metadata & Governance project Meta data highway
- 26. 1. Tink, a temporal graph analytics library for Apache Flink. 13-9 at 14:00 – Palais Atelier 2. Fast Data at ING – building a streaming data platform with Flink and Kafka 13-9 at 15:20 - Kesselhaus Drop me a note at Ferd.Scheepers@ing.com Or: @Ferdscheepers Before I sign off, there is more from ING tomorrow: 26