Reaching scale limits on a Hadoop platform: issues and errors created by speed and agility
Download as PPTX, PDF0 likes1,608 views
Santander UK is undergoing a significant transition towards becoming a data-driven organization, focusing on speed and agility through the use of advanced technologies like Hadoop and Spark. Over a two-year journey, they have scaled their data projects significantly while emphasizing automation and a cultural shift towards data democratization. Key challenges include managing system performance, ensuring security, and balancing innovation with governance as growth places demands on their data infrastructure.
Reaching scale limits on a Hadoop platform: issues and errors created by speed and agility
- 1. Reaching scale limits on a Hadoop platform. Lessons from a journey of speed and agility Data innovation Santander UK
- 2. Customer Technology Data People Nici Bullivant Head of Engineering Data innovation| Santander UK
- 3. Santander UK is a scale challenger 150 years of banking in our DNA. Culture of retail banking. Moderate risk. A very robust and traditional bank… with a culture of technology. 5th biggest bank in the UK. Growth by successful mergers. From 2004 to 2013, Santander bought Abbey, Bradford&Bingley, Alliance&Leicester and other smaller portfolios. Now transforming into a Data driven organisation.
- 5. Proof of Concept • 3 months • Test, find potential, identify limits, plan the future • Hadoop, Hive, Impala, Spark, Oozie, Hue, Solr T+3months
- 6. Analytics • Get a tool immediately • Data, business and technology in collaboration T+5months
- 7. Customer facing product - Spendlytics • The showcase: real time analytics for customers • Building, installing, breaking, fixing • Flume, Kafka, Hbase, APIs T+9months
- 8. Broadening • Early adopters • Feedback and improve • Open up for business • Establish the foundations for growth T+12months
- 9. Scaling up • Expansion of data: 10X; Projects: 12X; Users: 30X • Continuously accelerate by automating and reusing • Define value and build the operating model around it T+24months
- 10. Democratisation • Guided self service • Data is a public good • Love your data • Care for your data T+24months
- 11. The Data Driven Organisation • Scientific • Embedded • Real time • Cultural change T+?
- 13. {Our current scale} {3,500 tables regularly ingested, Over 20,000 tables} {200+ tables streaming from core banking} {3+ PB in total} {30+ products in Production} {1,000 users} {50 BI applications in development} {CDH 5.5.4 after upgrade from CDH 5.4.2 early 2017}
- 14. ACQUIRE EXPLOIT UNDERSTAND TraceDescribe Engines Visualise (Micro Strategy) Iterate (notebooks) Organise Analyse (SAS) Model (DataRobot) Ingest Certify Capture (CDC) Stream SERVE Interfaces New databases APIs MANAGE Access Monitor Support Evolve Operate Data from source systems Analyst Internal user Maintain the information at rest. STORE Process the data. COMPUTE Govern Discover Technology - Conceptual architecture Applications Data scientist
- 15. Limits to performance Metastore - too many partitions, and ungraceful stops Namenode - large memory footprint Small files, Joining files Navigator & Replication HBase
- 17. {But upgrading is a challenge in itself} {Duplicate environments} {Recompile and deploy} {Use upgrade to surface latent business products} {Automate, automate, automate} Lessons from our upgrade experience UPGRADE!
- 19. {Is cheap and is now mature} {Improvements in security, management, operation} {New processing frameworks: Spark, Flink…} {Streaming reaches maturity} {More and more integrated tools} {Going into Big data now could mean leapfrogging the early entrants}
- 20. {Incremental data} {Snapshots occupy too much data} {Incremental copies are more difficult to manage but make better use of space} {Incremental still run full scans in tables, unless you implement Change Data Capture} {…}
- 21. {Automate everything} {The only path to speed and scale is automation} {Architecture as a Service} {Test driven development, test automation} {CI/CD: Automation of deployment} {Compliance as a Service: Automation of documentation} {Machine learning for Data Governance}
- 22. Generate value quickly Identify use cases that are valuable, create learning and have a wow factor. As use cases increase, define products that need to remain and patterns that repeat. For those patterns, create services by using the learnings and tools delivered by the use cases.
- 23. And maintain the fresh water… Get original data Minimise data duplication and inconsistencies Speed of Innovation and Self-service overtakes governance Balance self service with self governance Cultural transformation slower than technology change
- 24. With value comes growth. Growth creates a challenge of scale. Scale requires planning and design. A robust foundation is necessary for a data platform to maintain agility. Make your foundation flexible too. Find a simple way to establish security, governance, architecture and automation.
- 25. Thank you… and get in touch nicolette.bullivant@santander.co.uk P.S. We are recruiting https://www.santandertechnology.co.uk/businessareas/data-innovation
Editor's Notes
- #3: Antonio has an extensive background in consultancy and financial services. With a strong trail leading a number of international transformation projects he integrated many legacy platforms and processes into the Santander strategic solution. Now he is the visionary leading Santander UK transformation into a Digital Data Driven organization, transforming the technology, the processes and the culture of the organization.
- #20: But what about launching your own new platform??? Now, that is a game changer… I believe innovators don’t set out to launch a platform… they may have a vision of a platform in their subconscious mind and, by launching their products advance on their vision… APIS MICROSERVICES SERVICE ORIENTED ARCHITECTURE
- #23: How do you integrate governance in continues delivery and scale it up (traditionally central team checkpoint controls) Traditional fight re customer number, Convince Risk, Finance, Marketing of sharing their customer view to understand the difference together – Consistency / Diversity understood / Find synergies The cop and the Villain of the movie
- #24: Also speed of growth caused chaos as knowledge did not spread fast enough Quick productionalization in innovation and self service skipping governance, then the controls are requested when there is a failure instead of being proactive, late to fix design if not involved in earlier phases Democratization – reduce complexity of access and new areas have more data and more opportunities but these new areas need to be educated to do the right thing (heavy regulations and responsibilities in financial services) Biggest challenge to transform culture “carrot and stick” – give them a benefit to WANT to collaborate – how to evidence benefits when you are starting