s2gx2015 who needs batch
Download as PPTX, PDF1 like710 views
This document discusses batch processing versus stream processing. It defines batch processing as processing bounded data sets without interruption, while stream processing handles unbounded asynchronous data. Common use cases like ETL, data ingestion, and connected devices are examined in terms of whether batch or stream processing is more appropriate. The document concludes that both approaches have merits, and combining them provides the most power, with stream processing platforms like Spring XD and Cloud Dataflow offering the best of both worlds.
s2gx2015 who needs batch
- 1. SPRINGONE2GX WASHINGTON, DC Unless otherwise indicated, these slides are © 2013-2015 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Who Needs Batch? By Michael Minella and Gunnar Hillert @michaelminella, @ghillert
- 3. Michael Minella Twitter: @michaelminella Podcast: http://javaOffHeap.com or @OffHeap Website: http://spring.io Gunnar Hillert Twitter: @ghillert Website: http://blog.hillert.com Conference: http://devnexus.com
- 6. W E H AV E A N ANNOUNCEMENT
- 7. WE HAVE A DISEASE
- 9. Shiny Object Syndrom SHINY OBJECT SYNDROME
- 11. ROAD TO RECOVERY
- 12. COOL KIDS ARE DOING STREAM PROCESSING
- 16. LET’S TAKE A CLOSER LOOK
- 19. LOOK AT COMMON USE CASES
- 20. CONCLUSIONS
- 22. A system for processing an unbounded set of data in an asynchronous manor.
- 24. DEPENDS ON WHO YOU ASK
- 25. “Batch … applications run … as special cases of stream processing applications” - Apache Flink Documentation
- 26. Batch processing is the processing of a bounded data set without interruption or interaction
- 27. KEY DIFFERENCES
- 35. JSR-352
- 36. WHY THE END OF BATCH?
- 37. LATENCY
- 38. WHAT DOES THE END OF BATCH MEAN?
- 39. LIMIT LATENCY
- 41. JSR-352
- 42. LOOK AT COMMON USE CASES
- 43. E.T.L.
- 44. DATABASE TO HDFS
- 45. DEMO
- 46. STREAMING CAN BE USEFUL IN INGESTION USE CASES
- 47. TWITTER TO HDFS
- 48. DATA SCIENCE
- 50. CONNECTED CAR
- 53. BREADTH AND ACCURACY DON’T MATCH BATCH ≠
- 58. DEMO
- 59. HOW IS THIS DONE VIA STREAMING?
- 62. PORT SCANNER
- 63. DEMO
- 64. STREAMING DOESN’T MAKE SENSE
- 65. STREAMING DOES MAKE SENSE
- 66. WHERE LATENCY IS A PRIORITY
- 69. BUT DOES NOT IN OTHER USE CASES
- 70. HIGHLY COMPLEX
- 71. ERROR HANDLING NOT AS ROBUST
- 72. DATA GUARANTEES NOT AS ROBUST
- 73. WHERE BATCH IS BETTER
- 74. FINITE DATASET
- 77. NOT EITHER OR
- 78. COMBINING THE TWO PROVIDES THE MOST POWER
- 79. E.T.L.v2
- 80. mailhttp mysql2hdfs Database to HDFS v2
- 81. DEMO
- 83. BATCH HAS LASTED BECAUSE IT MAKSE SENSE
- 84. SPRING XD/DATA FLOW BEST OF BOTH WORLDS
- 86. BEST BATCH OPTION ON THE JVM
- 88. Unless otherwise indicated, these slides are © 2013-2015 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 88 Check out Spring Cloud Data Flow on https://spring.io Apache Spark for Big Data Processing – Salon N-P Up next! High Performance Stream Processing – Salon N-P 8:30AM Tomorrow Spring Integration Extensions Ecosystem – Here! 12:45 Tomorrow Learn More. Stay Connected. @springcentral Spring.io/video