Spring Batch Performance Tuning
22 likes12,311 views
The document discusses Spring Batch and Spring Integration, focusing on performance tuning and scaling techniques for batch processing tasks. It highlights features such as transaction management, chunk-based processing, and various integration styles for handling large volumes of data. Additionally, it covers advanced topics like remote chunking, partitioning, and how Spring XD can be utilized for complex big data workflows.
Spring Batch Performance Tuning
- 1. Spring Batch Performance Tuning By Chris Schaefer & Gunnar Hillert © 2014 SpringOne 2GX. All rights reserved. Do not distribute without permission.
- 2. Agenda • Spring Batch • Spring Integration • Spring Batch Integration • Scaling Spring Batch • Spring XD 2
- 3. Spring Batch 3 http://projects.spring.io/spring-batch/
- 4. Batch processing ... is defined as the processing of data without interaction or interruption. 4 “ Michael T. Minella, Pro Spring Batch
- 5. Batch Jobs • Generally long-running • Non-interactive • Often include logic for handling errors and restartability options • Process large volumes of data • More than what may fit in memory or a single transaction 5
- 6. Batch and offline processing • Close of business processing • Order processing, Business reporting, Account reconciliation, Payroll • Import / export handling • a.k.a. ETL jobs (Extract-Transform-Load) • Data warehouse synchronization • Large-scale output jobs • Loyalty program emails, Bank statements • Hadoop job orchestration 6
- 7. Features • Transaction management • Chunk based processing • Schema and Java Config support • Annotations for callback type scenarios such as Listeners • Start/Restart/Skip capabilities • Based on the Spring framework • JSR 352: Batch Applications for the Java Platform 7
- 8. Concepts • Job • Step • Chunk • Item 8 Repeat | Retry | Skip | Restart
- 9. Chunk-Oriented Processing • Read data, optionally process and write out the “chunk” within a transaction boundary. 9
- 10. JobLauncher 10
- 11. ItemReaders and ItemWriters • Flat File • XML (StAX) • Multi-File Input • Database • JDBC, JPA/Hibernate, Stored Procedures, Spring Data • JMS • AMQP • Email • Implement your own... 11
- 12. Simple File Load Job 12
- 14. Spring Integration 14 http://projects.spring.io/spring-integration/
- 15. Integration Styles • File Transfer • Shared Database • Remoting • Messaging 15
- 16. Integration Styles • Business to Business Integration (B2B) • Inter Application Integration (EAI) • Intra Application Integration 16 JVM JVM EAI Core Messaging B2B External Business Partner
- 17. Common Patterns 17 Retrieve Parse Transform Transmit
- 18. Enterprise Integration Patterns • By Gregor Hohpe & Bobby Woolf • Published 2003 • Collection of well-known patterns • Icon library provided 18 http://www.eaipatterns.com/eaipatterns.html
- 19. Spring Integration provides an extension of the Spring programming model to support the well-known enterprise integration patterns. 19 “ Spring Integration Website
- 20. Adapters 20 AMQP/RabbitMQ AWS File/Resource FTP/FTPS/SFTP GemFire HTTP (REST) JDBC JMS JMX JPA MongoDB POP3/IMAP/SMTP Print Redis RMI RSS/Atom SMB Splunk Spring Application Events Stored Procedures TCP/UDP Twitter Web Services XMPP XPath XQuery ! Custom Adapters
- 21. Samples • https://github.com/spring-projects/spring-integration-samples • Contains 50 Samples and Applications • Several Categories: • Basic • Intermediate • Advanced • Applications 21
- 22. Spring Batch Integration 22
- 23. Launching batch jobs through messages • Event-Driven execution of the JobLauncher • Spring Integration retrieves the data (e.g. file system, FTP, ...) • Easy to support separate input sources simultaneously 23 D C FTP Inbound Channel Adapter JobLauncher Transformer File JobLaunchRequest
- 24. JobLaunchRequest 24 public class FileMessageToJobRequest {! private Job job;! private String fileParameterName;! ...! @Transformer! public JobLaunchRequest toRequest(Message<File> message) {! JobParametersBuilder jobParametersBuilder = new JobParametersBuilder();! jobParametersBuilder.addString(fileParameterName,! message.getPayload().getAbsolutePath());! return new JobLaunchRequest(job, jobParametersBuilder.toJobParameters());! }! }!
- 25. JobLaunchRequest 25 <batch-int:job-launching-gateway request-channel="requestChannel"! reply-channel="replyChannel"! job-launcher="jobLauncher"/>!
- 26. Get feedback with informational messages ! • Spring Batch provides support for listeners: • StepExecutionListener • ChunkListener • JobExecutionListener 26
- 27. Get feedback with informational messages 27 <batch:job id="importPayments"> ... <batch:listeners> <batch:listener ref="notificationExecutionsListener"/> </batch:listeners> </batch:job> ! <int:gateway id="notificationExecutionsListener" service-interface="o.s.batch.core.JobExecutionListener" default-request-channel="jobExecutions"/>
- 28. Launching and information messages demo in next section 28
- 29. Scaling Spring Batch 29
- 30. Scaling and externalizing batch process execution • Utilization of Spring Integration for multi process communication • Distribute complex processing • Single process o Multi-threaded steps o Parallel steps o Local partitioning • Multi process o Remote chunking o Remote partitioning • Asynchronous Item processing support • AsyncItemProcessor • AsyncItemWriter 30
- 31. Single Thread 31 Reader Item Result Gateway Output Input Processor Writer Item Result
- 32. Single Thread - Demo 32
- 33. Multi-threaded 33 • Simply add a TaskExecutor to your Tasklet configuration Reader Item Result Gateway Output Input Processor Writer Item Result
- 34. Multi-Threaded - Demo 34
- 35. Asynchronous Processors • AsyncItemProcessor • Dispatches ItemProcessor logic on new thread, returning a Future to the AsyncItemWriter • AsyncItemWriter • Writes the processed items after processing is complete 35
- 36. Asynchronous Processors - Demo 36
- 37. Remote Chunking 37 Step 2a ItemReader ItemProcessor ItemWriter Step 1 ItemReader ItemProcessor ItemWriter Step 2 ItemReader ItemWriter Step 3 ItemReader ItemProcessor ItemWriter Step 2b ItemReader ItemProcessor ItemWriter Step 2c ItemReader ItemProcessor ItemWriter
- 38. Remote Chunking - Demo 38
- 39. Remote Partitioning 39 Slave 1 ItemReader ItemProcessor ItemWriter Step 1 ItemReader ItemProcessor ItemWriter Master Step 3 ItemReader ItemProcessor ItemWriter Slave 2 ItemReader ItemProcessor ItemWriter Slave 3 ItemReader ItemProcessor ItemWriter Partitioner
- 40. Remote Partitioning - Demo 40
- 41. Demo - Launching via messages & informational messages 41 Does not provide scaling but demonstrates how launch job via messages and send information messages to integration points
- 42. Spring XD 42 http://projects.spring.io/spring-xd/
- 43. Tackling Big Data Complexity ! • Data Ingestion • Real-time Analytics • Workflow Orchestration • Data Export 43
- 44. Tackling Big Data Complexity cont. ! • Built on existing Spring assets • Spring Integration • Spring Batch • Spring Data • Spring Boot • Spring for Apache Hadoop • Spring Shell • Redis, GemFire, Hadoop 44
- 45. Data Ingestion Streams • DSL based on Unix pipes and filters syntax ! • Modules are parameterizable ! • Simple logic can be added via expressions or scripts 45 http | file twittersearch --query=spring | file --dir=/spring http | filter --expression=payload=='Spring' | hdfs
- 46. Hadoop workflow managed by Spring Batch • Reuse Batch infrastructure and features to manage Hadoop workflows • Job state management, launching, monitoring, restart/retry policies, etc. • Step can be any Hadoop job type or HDFS script • Can mix and match with other Batch readers/ writers, e.g. JDBC for import/export use-cases 46
- 47. Manage Batch Jobs with Spring XD 47
- 48. 48 Spring XD - Demo
- 49. Books 49
- 50. Learn More. Stay Connected. ! ! ! Demo code and slides: https://github.com/SpringOne2GX-2014/spring-batch-performance-tuning 50 THANK YOU!