Delivering Cloud Native Batch Solutions - Dodd Pfeffer
Download as PPTX, PDF1 like464 views
This document discusses using cloud native batch solutions for scheduled batch jobs and modernizing monolithic ETL workloads. It describes how batch applications on Pivotal Cloud Foundry could leverage Spring Batch, Spring Cloud Task, Spring Cloud Data Flow, and other Spring Cloud services for a cloud native approach to batch processing. Specific examples are provided for using these technologies for file ingest, predictive modeling, and the finance company Dragonstone.
Delivering Cloud Native Batch Solutions - Dodd Pfeffer
- 1. © Copyright 2019 Pivotal Software, Inc. All rights Reserved. Dodd Pfeffer Pivotal Platform Architect Cloud Native Batch
- 3. Use Cases 3
- 4. Modernize monolithic ETL workloads SQL scripts, stored procedures, and in-house bash scripts to cloud- native architecture. Small and incremental releases. Continuous delivery is the focus. 4
- 5. Scheduled batch-jobs Whether it is for predictive model training, massive file movement, or the classic data migration batch-jobs, they are typically schedule driven. 5
- 6. Dragonstone Finance ● Good DDD design approaches ● Internal ratings service ○ Influenced by a series of analyst evaluations provided via file integration from various sources ● Internal auditing function ○ Accepts trade data via file integration from various sources ○ Provides feedback to traders and trading system on healthiness of trades
- 7. What would a cloud native batch solution look like ● Batch apps run on ephemeral containers whose capacity is only consumed during batch execution ● Polling for new files delivered by data microservice pipelines via ingest streams ● Scheduler provided by the platform ● Batch app properties externally managed through config server ● Batch app external service dependency facilitated through service registry ● Cloud platform dynamically binds persistent volume-service to container during batch execution ● Database connection information dynamically injected into batch app environment ● Databases provisioned and managed by via cloud platform ● Batch app logs aggregated and stored on the platform, optional streamed to external log source ● Access to manage and monitor the batch applications consistent with other microservices on the platform
- 8. Dragonstone Finance ● Spring Batch ● Spring Cloud Task ● Spring Cloud Data Flow ● Spring Cloud Config Server ● Spring Cloud Service Registry ● Pivotal Application Services ● Volume Services for PCF ● MySql for PCF
- 9. File Ingest Reference Architecture https://content.pivotal.io/blog/need-24x7-etl-then-move-to-cloud-native-file-ingest-with-spring-cloud-data-flow
- 10. Spring Batch ● Lightweight framework ● Familiar spring interfaces ● Transaction management ● Chunk based processing ● Declarative I/O ● Start/Stop/Restart ● Retry/Skip https://docs.spring.io/spring-batch/4.1.x/reference/html/index-single.html
- 11. Spring Cloud Stream ● Stream is composed of long lived Boot Microservice apps ● Event driven based Microservices ● Loose coupling via pub/sub messaging ● Opinionated Middleware Primitives ● For integration and stream processing use-cases ● Each microservice app can be packaged as an uber-jar or as a Docker container ● Out-of-the-box apps: https://cloud.spring.io/spring-cloud-stream-app-starters/
- 12. Spring Cloud Task ● Tasks are short lived Boot Microservice apps ● System tracks invocations, exit-status ● Spring Batch Jobs are wrapped as Tasks ● Useful for ETL between databases and filesystems ● Each microservice app can be packaged as an uber-jar or as a Docker container ● Out-of-the-box apps: https://cloud.spring.io/spring-cloud-task-app-starters/
- 13. Spring Cloud Stream ● Spring Boot Application that stays on listening for a data source. ● Provides abstraction to a message broker implementation. Stream Versus Task Spring Cloud Task ● Spring Boot application that terminates its process when it has completed its task. ● Implements a CommandLineRunner bean ● May publish messages using Spring Cloud Stream. ● Maintains its own task repository to track task analytics
- 14. What is Spring Cloud Data Flow? Spring Cloud Data Flow is a Microservices toolkit for building data integration and real-time data processing pipelines. Pipelines consist of Spring Boot apps, using Spring Cloud Stream for events or Spring Cloud Task for batch processes. The Data Flow server provides interfaces to compose and deploy pipelines onto a modern platform like Cloud Foundry.
- 15. Cover w/ Image Spring Cloud Services for Pivotal Cloud Foundry ■ Pivotal has bundled a number of these OSS projects into Spring Cloud Services for PCF Service Registry Config Server Circuit Breaker Dashboard
- 16. Increase speed and deploy code to production thousands of times per month. Use PAS to run Java, .NET, and Node apps. Pivotal Application Service (PAS): A Runtime for Apps Best runtime for Spring and Spring Boot — Spring’s microservice patterns—and Spring Boot’s executable jars—are ready-made for PAS. Turnkey microservices operations and security — Spring Cloud Services brings microservices best practices to PAS. It includes Config Server, Service Registry, and Circuit Breaker Dashboard. A native Windows and .NET experience — Use PAS to run new apps built with .NET Core. Run your legacy .NET Framework apps on PAS too, using the .NET Hosted Web Core buildpack. Push applications to containers running on Windows Server 2016. Built for apps — PAS has everything to need to run apps. Buildpacks manage runtime dependencies; metrics, logging, and scaling are done for you. Multitenancy, and blue/green deployment patterns are built-in. Extend apps with a rich service catalog. Container-ready — PAS supports the OCI format for Docker images. Run platform-built and developer-built containers.
- 17. Let’s take a look
- 18. References ● 24x7 ETL / File Ingest Reference Architecture: https://content.pivotal.io/blog/need-24x7-etl-then- move-to-cloud-native-file-ingest-with-spring-cloud-data-flow ● Spring Batch Docs: https://docs.spring.io/spring-batch/4.1.x/reference/html/index-single.html ● Spring Cloud Data Flow Docs: https://docs.spring.io/spring-cloud- dataflow/docs/current/reference/htmlsingle/ ● Volume Services: https://docs.pivotal.io/pivotalcf/2-5/devguide/services/using-vol- services.html#create-nfs ● Demo Source Repo: https://github.com/doddatpivotal/dragonstone-finance
- 19. Register Today Early Bird ends 7/8 Use Discount Code S1P_Save200 October 7–10, 2019 Austin Convention Center and Save!
- 20. Transforming How The World Builds Software © Copyright 2017 Pivotal Software, Inc. All rights Reserved.Revised 1/25/19
Editor's Notes
- #2: This presentation introduces Pivotal, and our value prop, to a new audience. It’s targeted at executives and decision makers.
- #3: Tell the story of my design decisions: Two different runtime platforms Different ACLS to access and manage Teams to request resources Manage servers, i don’t want to do that
- #17: Pivotal Application Service (or PAS, formerly Elastic Runtime or ERT) sits at the core of what Pivotal offers. As Pivotal’s Platform as a Service (PaaS) offering, it delivers on the promises of Microservices & DevOps at scale in many ways: It provides the best place for Spring and Spring Boot applications to run. Many other popular languages are fully supported with native buildpacks - NodeJS, Python, PHP, Ruby, Go, .Net just to name a few. PAS was built for microservices, and natively integrates with many of the patterns that are necessary to use microservices at scale. Spring Cloud Services introduces a production-ready implementation of these patterns, and can be deployed on PAS quickly. PAS offers a native Windows experience that can run both new and legacy .NET applications, with the same platform-level features as Linux-based applications. We like to use the term polyglot to refer to multiple languages, and by running multiple OSes under the covers, PAS allows for true polyplatform and polyglot deployments. Surrounding every application and service is a full ecosystem of management, monitoring, logging, scaling, and related services. These enable you to go from simple applications deployed in one container to 100s or 1000s of applications and services working together to comprise real enterprise applications, at a significant savings in every IT cost center. And last, as part of supporting multiple ways to deploy and manage applications, we support Docker as a first-class citizen. You can easily run and create public and private images, hosted publicly or privately.