Building Data Streaming Platforms using OpenShift and Kafka

Editor's Notes

• #2: Cover example with no photos Please insert the other company’s logo in the same size as the Amadeus logo, with no background and reverse out colour
• #3: Divider example with no photo You can change the colour to Amadeus blue, Light Blue and Pale Blue To update the text, just click on the text To get more slides with this layout just copy this slide and paste
• #4: It is based on event streams, similar to RDBMS redo logs: Log is persistent – the events are kept even when consumed, ready to be replayed. The events are strongly sequenced by construction so that we have the absolute guarantee that they are consumed in the right order. The events are immutable meaning that they can be replayed any number of times we’ll get the same result (so a counter increment operation will never fire a “+1” event but rather the result of the operation, e.g. “4”, so that whatever the number of times you’ll replay the event you’ll always get “4”). If instead of writing directly to several data stores (as we are doing right now in Amadeus) the applications are logging their data changes as event in an event log we can do a lot of great things: We can synchronize in an eventually consistent manner several data stores, each with its own required data representation (while consistency is very hard to achieve directly from the application). We can migrate easily from one data store technology (e.g. memcached) to another (e.g. Couchbase) by just plugging in a new log subscriber. Once the new store is populated (by just waiting or replaying old events just to this store), we can decommission the old one. Cherry on the cake, we can write a business logic that will interpret those events and produce a new output stream as a result. By chaining several data processors like that we can build a streaming graph as exposed previously.
• #5: It is based on event streams, similar to RDBMS redo logs: Log is persistent – the events are kept even when consumed, ready to be replayed. The events are strongly sequenced by construction so that we have the absolute guarantee that they are consumed in the right order. The events are immutable meaning that they can be replayed any number of times we’ll get the same result (so a counter increment operation will never fire a “+1” event but rather the result of the operation, e.g. “4”, so that whatever the number of times you’ll replay the event you’ll always get “4”). If instead of writing directly to several data stores (as we are doing right now in Amadeus) the applications are logging their data changes as event in an event log we can do a lot of great things: We can synchronize in an eventually consistent manner several data stores, each with its own required data representation (while consistency is very hard to achieve directly from the application). We can migrate easily from one data store technology (e.g. memcached) to another (e.g. Couchbase) by just plugging in a new log subscriber. Once the new store is populated (by just waiting or replaying old events just to this store), we can decommission the old one. Cherry on the cake, we can write a business logic that will interpret those events and produce a new output stream as a result. By chaining several data processors like that we can build a streaming graph as exposed previously.
• #6: Loose coupling – straightforward Read & write performance: with a unique RDBMS we design our data model always with a trade off between read and write performances. As here the write is asynchronous, write cost is not on application side and each store can have a data model well adapted to the need (read or write performant). High scalability with parallelization and massive distribution of the load via micro-services spread in any number of nodes. Flexibility to improve the application by just plugging in new micro-services without impacting the existing code base. Easy to migrate from old to new micro-service or data stores (both can run in parallel). High auditability as each event is persistent, and high recovery as they can easily be replayed in case of error.
• #10: Divider example with no photo You can change the colour to Amadeus blue, Light Blue and Pale Blue To update the text, just click on the text To get more slides with this layout just copy this slide and paste
• #17: Divider example with no photo You can change the colour to Amadeus blue, Light Blue and Pale Blue To update the text, just click on the text To get more slides with this layout just copy this slide and paste
• #21: Backcover option 2