![16 © Cloudera, Inc. All rights reserved.
MOTIVATING EXAMPLE: WORKLOAD MANAGEMENT
Direct Execution
• Consider a simple two-stage workload:
1. Load CSV data into HDFS staging:
2. Run a Spark ETL job to load into Hive/Parquet:
RISK COMPONENTS
Metadata Driven Automation
Metadata
Data
Processing
Egress
Me
A
Notifications
Housekeeping
Data
Quality
Test
Automation
Ingress
15
© Cloudera, Inc. All rights r
RISK COMPONENTS
Metadata Driven Automation
Metadata
Data
Processing
Egress
Metrics
Alerts
Notifications
SLA
Management
Housekeeping
Data
Quality
Test
Automation
Ingress
hdfs dfs -put localfiles/*.csv staging
spark-submit
--class <main-class>
--master <master-url>
--deploy-mode <deploy-mode>
--conf <key>=<value>
... # other options
<application-jar>
[application-arguments]
LOAD
PROCESS
Resources
Time
Max](https://image.slidesharecdn.com/041230deutschebankcloudera-181118100001/85/Big-Data-LDN-2018-DEUTSCHE-BANK-THE-PATH-TO-AUTOMATION-IN-A-HIGHLY-REGULATED-SPACE-16-320.jpg)
Big Data LDN 2018: DEUTSCHE BANK: THE PATH TO AUTOMATION IN A HIGHLY REGULATED SPACE
- 1. THE PATH TO AUTOMATION IN A HIGHLY REGULATED SPACE Paul Wilkinson and Naveen Gupta
- 2. 2 © Cloudera, Inc. All rights reserved. SPEAKERS (1) Naveen Gupta Background: • Architect / Developer ~ 20 Years • Financial Services ~ 14 Years Areas of Interest: • Enterprise Risk and Data Governance • Regulatory Demand in Market Risk • Data Architecture • White-Label Product Development in Risk Professional Experience: • Financial Services, Software, Civil Aviation, Healthcare Senior Vice President Market Risk, Deutsche Bank
- 3. 3 © Cloudera, Inc. All rights reserved. SPEAKERS (2) Paul Wilkinson Background: • Architect / Developer ~ 15 Years • Big Data ~ 10 Years Areas of Interest: • Data Engineering at Scale • Platform Architecture • Application Development Specialist Professional Experience: • DB Resident Solution Architect ~ 3 Years • Financial Services, Cyber Security, Telco Principal Solution Architect Professional Services, Cloudera Author: Architecting Modern Data Platforms
- 4. © Cloudera, Inc. All rights reserved. INTRODUCTION
- 5. 5 © Cloudera, Inc. All rights reserved. AND FINANCIAL SERVICES 19 Banks “Too Big to Fail” (G-SIB*) 12 Banks with Risk Use Cases 10,000+ Nodes Deployed 5+ Years in Production (* Global, Systemically Important Bank)
- 6. 6 © Cloudera, Inc. All rights reserved. BIG DATA USE CASES IN FINANCIAL SERVICES Customer Journey/360 Risk & Compliance Financial Crime Customer Loyalty, Retention PersonalizationNext Best Offer Risk Analytics, Underwriting, Actuarial BASEL, BCBS-239, FRTB, IFRS-9 Risk Aggregation, Stress Testing AML, Payments Fraud Insider Threats Cyber Security Investment Research Portfolio Analytics, Robo- Advisors IoT, Blockchain, ML/AI Product & Service Efficiencies Modern Data Architecture ETL/ Warehouse Optimization, Active Archive, Storage Optimization & Embedded Analytics
- 7. 7 © Cloudera, Inc. All rights reserved. POST-CRISIS REGULATIONS AFFECTING RISK TECHNOLOGIES Each crisis leads to a desire to prevent future occurrences: • Increased stress testing, more scenarios (CCAR). • Historical simulation of VaR (HISTSIM) • Backtesting requirements (US Basel III) • Improved risk data, technology and process controls (BCBS-239). • Individual model regulatory review and back-testing (FRTB).
- 8. 8 © Cloudera, Inc. All rights reserved. REGULATORY IMPACT (1) Huge Storage and Processing Growth – FRTB Estimates Data Storage Impact Position Size Reg. 1Yr Size (TB) 5-Yr History (PBs) Reg. 1Yr Size (TB) 5-Yr History (PB's) 100,000 Positions Basel 2.5: (TB) 3 0.15 FRTB: (TB) 72 3.6 1 Million Positions Basel 2.5: (TB) 30 1.5 FRTB: (TB) 720 36 Computational Impact Position Size Reg. Type Computations Reg. Type Computations 100,000 Positions Basel 2.5: (TB) Scenario Valuations 50 Million (Daily) & 50 Million (Week) FRTB: (TB) Scenario Valuations 1.57 Billion (Daily) Approximate 20x increase in Historical Data Storage Requirements Approximate 30x increase in Computational Requirements
- 9. 9 © Cloudera, Inc. All rights reserved. REGULATORY IMPACT (2) BCBS-239: Principles for Effective Risk Data Aggregation and Risk Reporting • Overarching Principals • Governance • Data Architecture and IT Infrastructure • Risk Data Aggregation • Accuracy and Integrity • Completeness • Timeliness • Adaptability • Risk Reporting Practices • Accuracy • Comprehensiveness • Clarity • Frequency • Distribution Significant impact on system design
- 10. © Cloudera, Inc. All rights reserved. THE EARLY JOURNEY
- 11. 11 © Cloudera, Inc. All rights reserved. EARLY ADOPTION AT DEUTSCHE BANK Challenges and Successes People • Difficult to find talent • Most developers had a relational mind-set Suppliers • Few suppliers had capabilities Technology • Apache Hadoop & Spark Standalone – no YARN • Open source with no vendor support Successes! • Delivered portfolio stress testing! • Historical simulation ingestion • Platform gained wider acceptance • Still needed to build an enterprise-class capability • Beginning of engagement with Cloudera PS
- 12. 12 © Cloudera, Inc. All rights reserved. EARLY RISK PLATFORM POC ~ 5 YEARS AGO Stress Testing, Historical and Monte Carlo Simulation • Poor Performance and Scalability • Lots of resource contention • Inconsistent SLAs • Platform bugs • Slow time-to-market • Bespoke solutions • No component reusability • Automation • Manual QA reconciliation • Long release process • Security or Governance • Compliance issue • Unsupported Platform • Difficult to get issues resolved • Platform support used developer time • Lack of Production Sandbox Capability • Lack of resource management • Activity contended with production • Lack of Standardisation • Segregated models for each function • Implicit Data Glossary
- 13. © Cloudera, Inc. All rights reserved. MODERNISING RISK DATA ANALYSIS Architect: Naveen Gupta
- 14. 14 © Cloudera, Inc. All rights reserved. “A CENTRALLY-MANAGED, UNIFIED DATA PLATFORM THAT COORDINATES RISK PROCESSING.”
- 15. 15 © Cloudera, Inc. All rights reserved. RISK DATA SERVICES - VISION • Shorten time-to-market • For data acquisition, processing and distribution • Provide workflow management • Decouple processing to enable scheduling • Standardise information architecture • Provide monitoring & notifications • Dismantle Legacy Systems • Accelerate Strategic Delivery • Rollout Strategy • Decommission • Improve Efficiency • Process data with continuous context • Provide Standardised Data Models • Conceptual, logical and physical • Glossary for data discovery • Increased agility • Measure Data Quality • Record-level and dataset-level acceptance criteria • Automated data profiling • Quality assessment & reporting • Implement Data Lineage • Full traceability from acquisition to distribution • Change management • Automate Testing • Continuous integration • Automated QA reconciliation
- 16. 16 © Cloudera, Inc. All rights reserved. MOTIVATING EXAMPLE: WORKLOAD MANAGEMENT Direct Execution • Consider a simple two-stage workload: 1. Load CSV data into HDFS staging: 2. Run a Spark ETL job to load into Hive/Parquet: RISK COMPONENTS Metadata Driven Automation Metadata Data Processing Egress Me A Notifications Housekeeping Data Quality Test Automation Ingress 15 © Cloudera, Inc. All rights r RISK COMPONENTS Metadata Driven Automation Metadata Data Processing Egress Metrics Alerts Notifications SLA Management Housekeeping Data Quality Test Automation Ingress hdfs dfs -put localfiles/*.csv staging spark-submit --class <main-class> --master <master-url> --deploy-mode <deploy-mode> --conf <key>=<value> ... # other options <application-jar> [application-arguments] LOAD PROCESS Resources Time Max
- 17. 17 © Cloudera, Inc. All rights reserved. MOTIVATING EXAMPLE: WORKLOAD MANAGEMENT The Problem with Direct Execution • Users can execute workloads at arbitrary times • Without YARN, this can lead to platform contention and instability • No contextual understanding of the data / action • Did the process run at the right time? • Were the right number of results produced? • Are there downstream processes that should now be run? • The context is implicit • SLAs were difficult to guarantee to the business 15 RISK COMPONENTS Metadata Driven Automation Metadata Data Processing SLA Management Housekeeping Data Quality Test Automation Ingress 15 Metadata Driven Automation Metadata Processing Egress Metrics Alerts Notifications SLA Management Housekeeping Data Quality Test Automation Ingress LOAD PROCESS Resources Time Max LOAD PROCESS LOAD PROCESS Time Max LOAD PROCESS Resources
- 18. 18 © Cloudera, Inc. All rights reserved. WORKLOAD MANAGEMENT Alternative Approach: Managed Execution • What if we decoupled ingress and processing? • A workload manager could run processes intelligently… … but what else could it do? RISK COMPONENTS Metadata Driven Automation Metadata Data Processing Egress Me A Notifications Housekeeping Data Quality Test Automation Ingress Workload Manager PROCESS LOAD PROCESS Resources Time Max LOAD
- 19. 19 © Cloudera, Inc. All rights reserved. WORKLOAD MANAGEMENT Metadata-Driven Execution The platform: • Chooses when to run processes • SLA / QoS Management • Configures them at runtime • Monitors their execution • Alerts operators when needed • Notifies up- and down-stream systems • Performs follow-on work: • Data quality analysis • Completeness & accuracy • Retention & Replication to DR • Is complementary to YARN Resourcing • Provides a powerful data context Workflow Management Data Processing Egress Metrics Alerts Notifications SLA Management Metadata Sandbox State Management Data Management Data Quality Test Automation Workload Replay Ingress
- 20. 20 © Cloudera, Inc. All rights reserved. METADATA / DATASET ABSTRACTION The risk platform maintains metadata: • Datasets: Storage system, location, format, partitioning • Processes: Spark Scripts, HDFS / Hive actions • Reports: Impala SQL queries, Spark dataframe Many clients access data through REST: • Provides API stability • Hides implementation detail • Provides runtime format conversion • Enables direct HDFS access for bulk data egress Multiple storage adapters : • Available for Spark, Impala, Hive, Kafka, … Client Platform Impala fetch query de-abstract resultset render
- 21. 21 © Cloudera, Inc. All rights reserved. CURRENT RISK PLATFORM Stress Testing, Historical Simulation, FRTB, US Basel III, CCAR • High Performance and Scalability • YARN Resource Management • Very large dataset processing now possible • 10x - 20x faster based on use cases • Fast time-to-market • Metadata-driven framework • Component reusability • Impact analysis / workflow tracking • Automation • Fully automated QA analysis • Load and performance benchmarking • Strong Security and Governance • Supported Platform • Internal Managed Hadoop Service • 24 x 5 Support from Cloudera • Full audit compliance • Large Scale Deployment • Multiple petabytes • 1,000s cores • Production Sandbox Capability • Faster data exploration • Activity segregated from production • Standardisation • Built-in data lineage / glossary
- 22. © Cloudera, Inc. All rights reserved. THE NEXT STAGE OF THE JOURNEY…
- 23. 23 © Cloudera, Inc. All rights reserved. RISK PLATFORM: FUTURE WORK • Framework Enhancements • Robotic Process Automation (RPA) • Self-service access for end-users • Increasing Scale and Adoption • 15-20x data volume expected • Ever increasing regulatory demands • More Automation • Machine-learning for release management • End-to-end container-based testing • Data Science • Sandbox capability extended to DS users • Opening up greater access to historical data • CDSW now available in lab!
- 24. 24 © Cloudera, Inc. All rights reserved. WE’RE HIRING! • Looking for: • Architects • Data Engineers • Data Scientists • Developers • Get in touch! • Email: naveen-a.gupta@db.com • In Person: Come over and chat! db.com/careers
- 25. THANK YOU