Pivoting Spring XD to Spring Cloud Data Flow with Sabby Anandan
8 likes2,747 views
The document discusses the transition from Spring XD to Spring Cloud Data Flow, emphasizing the importance of microservices architecture and continuous delivery for improved business strategy and results. It outlines various challenges and tools related to building data-driven microservices, including Spring Cloud Stream and Spring Cloud Task for real-time data insights and batch processing. Additionally, it highlights the benefits of a cloud-native model for orchestrating composable data microservices with built-in scalability and monitoring capabilities.
![Kodiak: Turnkey Data Integration Platform Sep 2012
Apr 2013
Aug 2013
Jan 2014
Apr 2014
Nov 2014
Dec 2014
Jan 2015
Feb 2015
Batch Jobs (Gaps: Oozie; Flume; Sqoop)
PMML (closed-loop-analytics; IoT), YARN
Kafka and Sqoop Support
Spring XD on CloudFoundry (beta)
Spring XD Launch
Builds on Spring Boot
Spark and Python Support
Spark Streaming, RxJava, Reactor Support
Apr 2015 BDS [HAWQ, GPDB, GemFire, PHD 3.0/HDP]
Flo UI, HDP/Ambari, Lattice Integration Jun 2015
Nov 2015 Flo Composed Jobs/Workflows, Perf. Improvements](https://image.slidesharecdn.com/scdfmeetupfeb2016-160210060818/85/Pivoting-Spring-XD-to-Spring-Cloud-Data-Flow-with-Sabby-Anandan-55-320.jpg)
![Kodiak: Turnkey Data Integration Platform Sep 2012
Apr 2013
Aug 2013
Jan 2014
Apr 2014
Nov 2014
Dec 2014
Jan 2015
Feb 2015
Batch Jobs (Gaps: Oozie; Flume; Sqoop)
PMML (closed-loop-analytics; IoT), YARN
Kafka and Sqoop Support
Spring XD on CloudFoundry (beta)
Spring XD Launch
Builds on Spring Boot
Spark and Python Support
Spark Streaming, RxJava, Reactor Support
Apr 2015 BDS [HAWQ, GPDB, GemFire, PHD 3.0/HDP]
Flo UI, HDP/Ambari, Lattice Integration Jun 2015
Nov 2015 Flo Composed Jobs/Workflows, Perf. Improvements](https://image.slidesharecdn.com/scdfmeetupfeb2016-160210060818/85/Pivoting-Spring-XD-to-Spring-Cloud-Data-Flow-with-Sabby-Anandan-56-320.jpg)
![Orchestrate Composable
Data Microservices
HOST
Cloud Foundry YARN Local
Spring Cloud Data Flow
µPCF
Spring Cloud Stream/Task AppsSpring Cloud Stream Binders [Rabbit, Kafka, Redis]](https://image.slidesharecdn.com/scdfmeetupfeb2016-160210060818/85/Pivoting-Spring-XD-to-Spring-Cloud-Data-Flow-with-Sabby-Anandan-70-320.jpg)
Pivoting Spring XD to Spring Cloud Data Flow with Sabby Anandan
- 1. Pivoting Spring XD to Spring Cloud Data Flow Sabby Anandan | Product Manager | @sabbyanandan
- 2. business premise microservices what it means for data tools / frameworks what we learned how it all comes together
- 4. Sense of Urgency OR Need for Momentum?
- 9. frequent roll out updates, even multiple times a day gives you more ways to maintain availability reduce risks allows you to make sound technology choices gives your team more velocity
- 10. microservices architecture business strategy business results devops culture continuous delivery Casey West Principal Technologist, Cloud Foundry
- 11. “how do we get there?”
- 13. Microservices architecture composed of loosely coupled elements that have bounded contexts Adrian Cockcroft Battery Ventures previously: Cloud Architect at Netflix
- 14. “what tools can I use?”
- 15. Spring
- 17. BUILDING ADAPTIVE APPLICATIONS IS HARD built on Cloud Foundry code will be open sourced. Demo: Spring Boot
- 18. Emerging Challenges • separate deployable units (registry, discovery, versioning) • reliable operationalization (fault tolerance) • too many moving parts (coordination, routing, load balancing)
- 19. Spring
- 20. Spring Cloud
- 21. Spring Cloud Hystrix Eureka Ribbon Zuul Feign Service Discovery Config Server Control Bus Zookeeper Distributed TracingConfig Server
- 23. OK, I get it.. ..what’s specical about microservices?
- 24. Microservices a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms, often an HTTP resource API James Lewis and Martin Fowler ThoughtWorks
- 25. Microservices a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms, often an an HTTP resource API
- 26. OK, I get it.. ..what if I have a lot of data?
- 27. every software company cares about data
- 28. Data-driven Microservices composition of small standalone executable applications communicating over messaging middleware Mark Fisher and Mark Pollack Spring/Pivotal
- 29. Data-driven Microservices composition of small standalone executable applications communicating over messaging middleware operates by consuming from and producing to message queues and data stores
- 30. “i need real-time insights from data”
- 31. Spring
- 32. Developer Toolkits Spring Integration enables lightweight messaging within Spring- based applications and supports integration with external systems Spring Cloud Stream enables you to create powerful distributed streaming data applications with a minimal amount of coding
- 34. http | transform | log
- 35. | = ? http | transform | log
- 36. | = Binder
- 37. Binders
- 38. BUILDING ADAPTIVE APPLICATIONS IS HARD built on Cloud Foundry code will be open sourced. Demo: Spring Cloud Stream
- 39. @EnableBinding(Source.class) Discover Output Channel Discover Binder Discover Type Converter Bind Producer Auto Configure $ java -jar http-source.jar
- 40. $ java -jar transform-processor.jar @EnableBinding(Processor.class) Discover Input and Output Channels Discover Binder Discover Type Converter Bind Producer and Consumer Auto Configure
- 41. @EnableBinding(Sink.class) Discover Input Channel Discover Binder Discover Type Converter Bind Consumer Auto Configure $ java -jar log-sink.jar
- 42. “i have large volumes of asynchronous data systems”
- 43. Spring
- 45. 10 11 12 13 14 15 16 17 18 19.. .. 10 11 12 13 14 15 16 17 18 19.. .. i0 i1 12 17
- 46. “i need offline data analysis”
- 47. Spring
- 48. Batch-processing Microservices Spring Batch enables the development of robust batch applications vital for the daily operations of enterprise systems Spring Cloud Task enables you to develop and run short-lived executable data applications locally or in the cloud
- 49. @EnableTask Discover DataSource Bind DataSource Auto Configure $ java -jar timestamp-task.jar
- 50. BUILDING ADAPTIVE APPLICATIONS IS HARD built on Cloud Foundry code will be open sourced. Recap
- 51. I need real-time insights from data I need offline data analysis Spring Cloud Stream Spring Cloud Task Stream Batch
- 53. Spring
- 54. Spring XD
- 55. Kodiak: Turnkey Data Integration Platform Sep 2012 Apr 2013 Aug 2013 Jan 2014 Apr 2014 Nov 2014 Dec 2014 Jan 2015 Feb 2015 Batch Jobs (Gaps: Oozie; Flume; Sqoop) PMML (closed-loop-analytics; IoT), YARN Kafka and Sqoop Support Spring XD on CloudFoundry (beta) Spring XD Launch Builds on Spring Boot Spark and Python Support Spark Streaming, RxJava, Reactor Support Apr 2015 BDS [HAWQ, GPDB, GemFire, PHD 3.0/HDP] Flo UI, HDP/Ambari, Lattice Integration Jun 2015 Nov 2015 Flo Composed Jobs/Workflows, Perf. Improvements
- 56. Kodiak: Turnkey Data Integration Platform Sep 2012 Apr 2013 Aug 2013 Jan 2014 Apr 2014 Nov 2014 Dec 2014 Jan 2015 Feb 2015 Batch Jobs (Gaps: Oozie; Flume; Sqoop) PMML (closed-loop-analytics; IoT), YARN Kafka and Sqoop Support Spring XD on CloudFoundry (beta) Spring XD Launch Builds on Spring Boot Spark and Python Support Spark Streaming, RxJava, Reactor Support Apr 2015 BDS [HAWQ, GPDB, GemFire, PHD 3.0/HDP] Flo UI, HDP/Ambari, Lattice Integration Jun 2015 Nov 2015 Flo Composed Jobs/Workflows, Perf. Improvements
- 57. Not Quiet There • couldn’t scale without undeploying running stream or batch pipelines • couldn’t do rolling upgrades • couldn’t do continuous deployments
- 58. longer running services short lived processes Data Microservices Architecture Stream Batch
- 59. Spring XD
- 60. Spring Cloud Data Flow
- 61. Spring Cloud Data Flow is a cloud native programming and operating model for composable data microservices on a structured platform
- 62. Spring Cloud Data Flow is a cloud native programming and operating model for composable data microservices on a structured platform
- 63. Spring Cloud Data Flow is a cloud native programming and operating model for composable data microservices on a structured platform @EnableBinding(Source) class Greeter { @InboundChannelAdapter(Source.OUTPUT) String greet() { "hello world" } } canary deployments continuous delivery monitoring @EnableBinding(Source.class) @EnableBinding(Processor.class) @EnableBinding(Sink.class)
- 64. Spring Cloud Data Flow is a cloud native programming and operating model for composable data microservices on a structured platform http transform log job foo < bar || baz & jaz > bye Streams Jobs foo bar jaz baz bye
- 65. Spring Cloud Data Flow is a cloud native programming and operating model for composable data microservices on a structured platform Private and Public PaaS Polyglot Multiple Platforms Service Marketplace Direct Access to Containers Metrics, Logging, Monitoring Dynamic Scaling and Resource Allocation Integrated Security and Auditing
- 66. Spring Cloud Data Flow is a cloud native programming and operating model for composable data microservices on a structured platform Private and Public PaaS Polyglot Multiple Platforms Service Marketplace Direct Access to Containers Metrics, Logging, Monitoring Dynamic Scaling and Resource Allocation Integrated Security and Auditing
- 67. Spring Cloud Data Flow is a cloud native programming and operating model for composable data microservices on a structured platform …… Polyglot Multiple Platforms …… Direct Access to Containers Metrics, Logging, Monitoring Dynamic Scaling and Resource Allocation Integrated Security and Auditing YARN
- 68. XD Modules XD Admin XD Containers ZooKeeper ZooKeeper Admin / Flo UI Shell CURL Database
- 69. Admin Admin / Flo UI Shell CURL ??X YARN Stream/Task Boot Apps
- 70. Orchestrate Composable Data Microservices HOST Cloud Foundry YARN Local Spring Cloud Data Flow µPCF Spring Cloud Stream/Task AppsSpring Cloud Stream Binders [Rabbit, Kafka, Redis]
- 71. BUILDING ADAPTIVE APPLICATIONS IS HARD built on Cloud Foundry code will be open sourced. Demo: Spring Cloud Data Flow
- 72. cf push • here is my source code • run it on the cloud for me • i do not care how
- 73. yarn push • here is my source code • run it on the hadoop for me • i do not care how
- 74. “simple things should be simple, complex things should be possible” - alan kay
- 75. Resources Spring Cloud Data Flow: http://cloud.spring.io/spring- cloud-dataflow/ Spring Cloud Data Flow Samples: https://github.com/ spring-cloud/spring-cloud-dataflow-samples Spring Cloud Stream: http://cloud.spring.io/spring-cloud- stream/ Spring Cloud Task: http://cloud.spring.io/spring-cloud-task/