An Architect's guide to real time big data systems
Download as PPTX, PDF3 likes861 views
This document is a guide discussing the architecture and implementation of real-time big data systems, focusing on concepts like streaming data, partitioned parallel processing, and analytics. It highlights tools such as IBM Infosphere Streams and Apache Storms for managing data pipelines and processing. The guide is intended for professionals in telecom and data architecture, specifically detailing application development and execution units.
An Architect's guide to real time big data systems
- 1. An Architect's Guide to Building Real Time Big Data Systems Raja SP 10 July 2014, Singapore Lead Architect & Head of Products
- 2. < Real Time > Big Data WHY WHAT HOW
- 3. < Real Time > Big Data WHY WHAT HOW
- 4. What is the right time to shoot me ?
- 5. There is a rhythm in the universe
- 6. Telecom Marketing Scenario Cell Utilisation is Low In a Geo-Fence High Balance Frequent Visitor High Data User in the Past
- 7. What is out there? Square Kilometers of Arrays Tens of Thousands of Antennae Terabits of Data
- 9. < Real Time > Big Data WHY WHAT HOW
- 10. Partitioned Parallel Processing TASK TASK TASK DATAi DATAj DATAk Pipelined Parallel Processing DATA TASK i TASK j TASK k TASKDATA Hybrid Parallel Processing DATA TASK i TASKj TASK mTASK k TASK l
- 11. TASKDATA Should Data go to Tasks? Or Tasks go to Data?
- 12. DATATASK TASK TASK TASK TASK TASK Static Data / Data at Rest DATA DATA DATA TASK DATA DATA DATA Streaming Data / Data in Motion
- 13. Streaming Data / Data in Motion Analytics
- 14. The classic “Word Count” (Stream Computing Version) Counter Counter Java Python Lisp Python Java C++ Counter Python Python Python 2 Token Splitter Sink
- 15. Stream Computing Programming Constructs Stream Tuple Operator / Bolt Counter Counter Java Python Lisp Python Java C++ Counter Python Python Python 2 Token Splitter Sink
- 16. Operator Source Operator Sink Operator IBM Infosphere Streams Apache Storms Bolt Spout ------- Composite Topology
- 17. Composite WordCountApp { Graph Stream< rstring sentence > Sentence = FileSource() {} Stream< rstring word > Word = Split( Sentence ) {} Stream< rstring word, int count > Counts = Count( Word ) {} } Source Split Count IBM Infosphere Streams Sentence Word Counts
- 18. Apache Storms TopologyBuilder builder = new TopologyBuilder(); builder.setSpout( ”Source", new RandomSentenceSpout(), 5 ); builder.setBolt( ”Split", new SplitSentence(), 8).shuffleGrouping( "Source” ); builder.setBolt( ”Count", new WordCount(), 12).fieldsGrouping( ”Split", new Fields( "word” )); Source Split Count
- 19. IBM Infosphere Streams – Some Operators Functor Perform tuple-level manipulations (~250 functions) Filter Remove some tuples from a stream Aggregate Group and summarize incoming tuples Sort Impose an order on incoming tuples in a stream Join Correlate two streams Punctor Insert window punctuation markers into a stream
- 20. IBM Infosphere Streams – Some Operators (continued) Barrier Synchronize tuples from sequence-correlated streams Pair Group tuples from multiple streams of same type Split Forward tuples to output streams based on a predicate ThreadedSplit Distribute tuples over output streams by availability Union Construct an output tuple from each input tuple DeDuplicate Suppress duplicate tuples seen within a given time period
- 21. DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA Stream Window Aggregate Sort Join
- 22. < Real Time > Big Data WHY WHAT HOW
- 24. Apache Storms RunTime Components IBM Infosphere Streams Instance Management Host Application Host Nimbus ZooKeeper Node 1 Node 2 Node 3 Cluste r
- 25. Apache Storms Application Deployment Units Instance Management Host Application Host 1 Processin g Element 1 Processin g Element 2 Cluste r Management Node (Nimbus) Node 1 Worker 1 Worker 2 Executor IBM Infosphere Streams Executo r Executo r ZooKeeper Node
- 26. High Availability & Adaptability Optimizing scheduler assigns jobs to nodes, and continually manages resource allocation Apache StormsIBM Infosphere Streams
- 27. High Availability & Adaptability Apache StormsIBM Infosphere Streams Dynamically add Nodes and Jobs
- 28. High Availability & Adaptability Apache StormsIBM Infosphere Streams Execution Units on Failed Nodes can be moved automatically with communications re- routed
- 29. Topic: Organized by UNICOM Trainings & Seminars Pvt. Ltd. contact@unicomlearning.com DEMO
- 30. Topic: Organized by UNICOM Trainings & Seminars Pvt. Ltd. contact@unicomlearning.com Speaker name: Raja SP Email ID: raja@knowesis.com Thank You
Editor's Notes
- #3: Enough chaos What – architectural thinking, programming concepts. Stream, Storm – map/reduce idea comes from lisp (1958). The 80’s game Can’t roll sleeves and deploy a 1000 node system
- #5: Option 1 – I am here you are pointing your gun to me. Will you pull the trigger right now? OR Option 2 – Wait until 3 hours after I left this place and THEN pull the trigger?
- #6: Wife cooks rearely….. I Thank god for that….. ½ km Spin Speed 30KM orbit speed
- #8: Radio Astronomy Tyco Brahe Uppsala University and the LOFAR Outrigger In Scandinavia (LOIS )
- #9: NSA breakout – prism, snowden Torture the data and it will confess to anything. Fallacy – Endogeneity Big Data has arrvied but not big Analytics – Tim Harford – the undercover economist – Financial Times
- #11: Singlish – sequential process – until cows come home oredy Shared nothing data Divide Data – Example – Calculating tax for all singaporean. Work hard and earn less group
- #13: Hadoop – Map Reduce Stream Computing
- #14: 13
- #15: Compare with map reduce Splitter heuristics continuous running streams – transient counts… sorts aggregates… windows A man cannot take bath in the same river twice
- #16: Tuple – composite of fields. Tuple Schema
- #17: 2 Popular frameworks
- #19: InputDeclarer
- #20: Relational Operators
- #21: Utility Operators
- #22: Tumbling Windows Sliding Windows
- #25: Describe the components Describe how they are deployed