Introducing Events and Stream Processing into Nationwide Building Society

Editor's Notes

• #3: Good morning all and thanks for joining this web cast I’m Rob Jackson HO Application Architecture for Nationwide and also from Nationwide we have Pete Cracknell who’ll be doing a demo for you today. Today we’re going to talk to you about an architecture we’ve called the “Speed Layer”. You might be familiar with the concept of a Speed Layer from Lambda Architectures and the world of Data architecture, but this isn’t that, it’s just a name that stuck, so sorry for any confusion we’ve caused with the name… I talk to you about the reasons why we’re doing this architecture, contrast it with our current state architecture Describe how we can consume data from the Speed Layer A bit about how it’s deployed Where we’re heading next The best bit is the demo and then we’ll do a Q&A with Tim (Vincent) from Confluent. I hope that’s ok!
• #4: Formed in 1884 Renamed to the name we are now in 1970 Worlds largest building society In the UK we’re a major provider of mortgages, loans, etc. Launched our first IB in 1997 and first or second Recently announced a large investment of 4.1bn What I’m talking to you about today forms an important part of that investment
• #5: I think the main headline for why we’re doing the speed layer is “digital disruption” Some might not see Open Banking as digital disruption as it’s a regulatory requirement However, it means we have to expose our data through APIs, and if we don’t offer good digital services through our own apps, customers will use other banks and organisations apps that use our APIs to disintermediate us. The other reason to mention Open Banking is that it was the catalyst to work on the Speed Layer. OB, had the potential for high and unpredictable read volumes along with stringent requirements for availability. We knew the other CMA9 were building OB with similar data caches to protect their core SORs from this load and we intended to do the same. I’m sure you’ll recognise the other headings there: higher volumes, 24x7 expectations, people expecting to use their data in new ways, e.g., how easy is it to search your emails vs. your bank’s transaction history? The final point is that, despite all the good work we do with them, our core ledgers do not make it easy to use data in new ways, aggregate multiple data sources, push events to customers and scale cost effectively.
• #6: Speed Layer one of the answers to Digital Disruption… First looking at the definition: It’s a source of data that can be queried for a near real-time copy of mainframe data. On top of that it introduces event sourcing and stream processing into the society. It’s built using modern technologies: Confluent Kafka, MongoDB, Microservices, OpenShift and will initially be deployed on-prem, but is very much an enabler for cloud – which I’ll come back to later. The 4 key characteristics: Resilient: the application is designed to tolerate infrastructure failure with build in data redundancy, horizonal scaling and automated recovery. Fast and agile: once we’ve extracted data from SORs, we can allow consumers to join, aggregate, structure, query and search data in ways the SORs do not easily allow. Rich Data set: Allows is to enrich SOR data with analytics, 3rd party sources, etc. And Scalability: scalability is designed in to the technologies we’re using. Kafka and MongoDB are both heavility used in internet scale deployments. My favourite statistic for Kafka is that Ali Baba use it to source events at a peak rate of 425 million TPS. A big number for us is a small proportion of that.
• #7: This isn’t really our current state, it’s just a representative sample of one small part of our estate and fairly common. But it allows me to describe a fairly normal transaction path A REST/http request for some data comes in from a device on the internet That hits an API Gateway in our datacentre, that then makes onward http requests until it eventually finds it’s way to the data in the mainframe If any of those layers is unresponsive the request will time out And of course, if the mainframe is not available, it doesn’t work at all. Thankfully, that doesn’t happen very often. If we want to move any of those components out into the cloud, that’s ok, but they still have to call back into our data centre to get to the mainframe. So, it’s not wrong and it’s how write requests will continue, perhaps with a simplified estate, fewer layers and modern technologies. However, for read requests, we can do something different.
• #8: This shows how Speed layer for reads and event sourcing will sit alongside our enterprise middleware A write comes into the SOR by existing means: batch, middleware, legacy services, payments gateway, whatever… It’s picked up by CDC and pushed into Kafka where it’s processed and stored before being materialised, in our case that’s MongoDB. Using this pattern, read requests are removed from our SORs or even our Data Centres, replicated to where it’s needed and materialised to requirements. Going into that in a bit more detail…
• #9: I’ll just talk you through the data flow… Step 1 is CDC on the mainframe. Of course, this enables multiple data sources, for example, batch files using Kafka Connect, applications creating events, but for us right now, it’s Change Data Capture on our Mainframes. So that’s how it works, next we’ll look at how consumers use it.
• #10: These show the ways in which we can consume data from the speed layer There are cases for all of these, but I’m very much looking forward to seeing the first 2 come to fruition Event driven – consumers subscribe to topics and act on events. Requst driven – data is materialised to requirements Functional – these are our cored shared services we expect to be re-used.
• #11: This shows how SL is an enabler for cloud and a good place to describe how resilience is baked into the architecture. Pete will show this for real during the demo.
• #12: Slide 10
• #13: Slide 11 We’re now getting into next steps We’re about to embark on a streaming assessment with Confluent’s help. We at around step 3, we’re using kafka, streaming, in SL, Mortgage and payments, but we’re currently doing things slightly differently on different platforms We want to look at new use cases, new demand and what capabilities we need to create to support that demand. This will feed into the various roadmaps, including the SI squad but also the IT Strategy.
• #14: Final slide before questions.
• #15: Pete was heading up the architectural proving team when we did this I approached him with an architecture I wanted to prove and I think Pete’s approach was to try to break it. He’ll let you know how he got on in the Q&A He can also cover the alternatives we looked at for stream processing and maybe some of the stuff we learnt along the way.