Journey to Big Data: Main Issues, Solutions, Benefits
1 like265 views
The document discusses the challenges, solutions, and benefits of big data in corporate and investment banking, specifically highlighting Santander's infrastructure and regulatory requirements. It describes the complexities of integrating over 1000 systems while maintaining data integrity and organization through ontologies and views. Additionally, it outlines the architecture and design patterns employed to manage data effectively, ensuring compliance and facilitating business operations.
Journey to Big Data: Main Issues, Solutions, Benefits
- 1. A journey to Big Data Main Challenges, Solutions and Benefits Corporate & Investment Banking October 2018
- 2. Who We Are? 2 Santander Highlights Total assets (EUR trillion) Gross cutormer loans (EUR billion excluding reverse repos) Customer deposits + mutual funds (EUR billion excluding repos) Branches 2017 Attributable profit (EUR million) H1’18 Attributable profit (EUR million) Market capitalisation (EUR billion; 26-06-18) People (headcount) Customers (millions) Shareholders (millions) Communities (million people helped in 2017) 1.43 863 895 13,482 6,619 3,752 74 200,961 140 4.2 2.1
- 3. Corporate & Investment Banking Business 3 Few contracts – Big profits Big Clients Details makes the difference Strong Competence Adapt to client systems Ad hoc integration Integration layer becomes huge Lot of middleware Big Client Big Client Big Client Big Client Big Client Big Client Big Client Big Client Big Client Big Client
- 4. Legacy Systems Challenges 4 Regulation Strong Regulation We cannot simply integrate systems in a row. We need to consolidate data for regulators Systems Lot of Systems There are more than 1000 different applications installed on Santander Corporate & Investment Banking Auditors Auditors are Welcome We need our systems ready to be audited at any time. Presence in Many Countries Same pattern is repeated on each country Countries
- 5. Big Data. Data Organization 5 Raw Data Data is stored ‘as is’ without any modification. It is required all data to have its own metadata Landing Data Ontologies Data is grouped according to functional criteria (ontologies). Data is consolidated to eliminate duplicates Common Business Views Is where applications access and process the data. It is not a copy but a view of the common layer Business Applications
- 6. Finland Architecture. A Thousand Lakes 6 API S3 Landing (RAW) Common (Harmonized) Business (Consolidated & Views) CDO BATCH CLUSTER A worker worker worker worker Tools BATCH CLUSTER B worker worker worker worker Tools OTHER BATCH CLUSTER C worker worker worker worker Tools
- 7. Finland Architecture. On Demand Evolution 7 API S3 Landing (RAW) Common (Harmonized) Business (Consolidated & Views) CDO ON DEMAND INSTANTIATION worker worker worker Datalake Distribution Tools worker worker worker Datalake Distribution Tools PROCESING AND SAVING DATA worker worker worker Datalake Distribution Tools FREE RESOURCES
- 8. Online Cluster Architecture 8 New Applications Legacy Applications Online Cache Business Rules Visualization Spring Data Flow Manual Orchestation Automatic Choreography REST API
- 9. Online Cluster. Design Pattern 9 Source Stream Online Cache Business Rules AVRO AVRO IDX AVRO IDX AVRO IDX AVRO AVRO Schema Registry AVRO AVRO IDX
- 10. Complete Finland Architecture 10 API S3 Landing (RAW) Common (Harmonized) Business (Consolidated & Views) CDO ON PREMISE BATCH CLUSTER worker worker worker Datalake Distribution Tools ON DEMAND BATCH CLUSTER Online Cache Business Rules Visualization Spring Data Flow Manual Orchestation Automatic Choreography REST API ONLINE CLUSTER worker worker worker worker Tools Datalake Distribution
- 11. Our purpose is to help people and business prosper. Our culture is based on believing that everything we do should be: Thank You.