Spring batch for large enterprises operations
14 likes2,514 views
Spring Batch is a framework for writing batch applications in Java. It was designed to be integrated with Spring and supports features like dependency injection, transactions, and job processing. The document discusses two examples of large enterprises using Spring Batch: a bank undergoing a migration to a new mid-level system and an energy company with multiple software vendors. It describes how Spring Batch was used to integrate batch jobs, monitor executions, and facilitate the migration process at both companies.
![Integration
Corporate batch launcher
ClassPathXmlApplicationContext sbc = BatchApplicationContextUtils.createApplicationContext(properties);
JobLauncher launcher = (JobLauncher) sbc.getBean("jobLauncher");
Job job = (Job) sbc.getBean(this.jobId);
initBatchContext(jobParameters, appParams, jobParam, sbc);
public static ClassPathXmlApplicationContext createApplicationContext(Properties credentials)
throws IOException {
String[] contextFiles = new String[]{
"classpath*:batch/arch/internal/beans/-batch-arch*.xml",
"classpath*:batch/app/**/-batch-app*.xml"
};
ClassPathXmlApplicationContext sbc = new ClassPathXmlApplicationContext(contextFiles, false);
PropertyPlaceholderConfigurer ppc = new PropertyPlaceholderConfigurer();
ppc.setSystemPropertiesMode(PropertyPlaceholderConfigurer.SYSTEM_PROPERTIES_MODE_NEVER);
ppc.setIgnoreUnresolvablePlaceholders(true);
ppc.setProperties(credentials);
sbc.addBeanFactoryPostProcessor(ppc);
sbc.refresh();
return sbc;
}
protected void initBatchContext(JobParameters jobParams, String[] appParams, String[] jobParam,
ClassPathXmlApplicationContext xmlcontext) {
BatchContext.reset();
BatchContext bc = BatchContext.getInstance();
BatchParams bp = new BatchParams();
bp.setJobName(this.jobId);
bp.setInstanceId(jobParams.getString(BatchLauncher.ArchParamKey.INSTANCIA.toString()));
bp.setScheduleTimestamp(jobParams.getDate(BatchLauncher.ArchParamKey.SCHED_TS.toString()));
bp.setAppParams(Arrays.asList(appParams));
bp.setJobParam(Arrays.asList(jobParam));
bp.setJobentorno(jobParams.getString(BatchLauncher.ArchParamKey.ENTORNO.toString()));
bp.setJobappmvs(jobParams.getString(BatchLauncher.ArchParamKey.APPMVS.toString()));
bc.setBatchParameters(bp);
bc.setxmlContext(xmlcontext);
}](https://image.slidesharecdn.com/springbatchforlargeenterprisesoperations20150429-150527142845-lva1-app6891/85/Spring-batch-for-large-enterprises-operations-24-320.jpg)
Spring batch for large enterprises operations
- 1. Spring Batch for large Enterprises operations Ignasi Gonzalez @ignasi15g
- 2. Let me do some initial questions
- 3. Which is the concept?
- 6. Agenda • Batch at Java and Spring • Usage of Spring Batch at a bank • Usage of Spring Batch at an energy company
- 7. Batch Structure JSR 352: Batch Applications for the Java Platform (May 24th, 2013)
- 8. Other Concepts Batchlet Chunk type Step Batchlet type Step Job Execution
- 9. JSR 352 & Spring Batch JSR – 352 • Define an API. • Provide an SDL (JSL) • Provide a contract. • Dependency injection is optional Spring Batch • Define an API. • Provide an SDL. (JSL) • Provide a contract. • Provide implementation for Readers and Writers. • Provide specific batchlet implementations. • Provide JobRepository implementation. • From Spring 3.0 support JSR-352.
- 11. Spring Batch Data Model http://docs.spring.io/spring-batch/trunk/reference/html/metaDataSchema.html
- 12. Spring Batch Data Model http://docs.spring.io/spring-batch/trunk/reference/html/metaDataSchema.html BATCH_JOB_INSTANCE
- 13. Spring Batch Data Model http://docs.spring.io/spring-batch/trunk/reference/html/metaDataSchema.html BATCH_JOB_EXECUTION
- 14. Spring Batch Data Model http://docs.spring.io/spring-batch/trunk/reference/html/metaDataSchema.html BATCH_JOB_EXECUTION_PARAMS
- 15. Spring Batch Data Model http://docs.spring.io/spring-batch/trunk/reference/html/metaDataSchema.html BATCH_STEP_EXECUTION
- 16. Spring Batch Data Model http://docs.spring.io/spring-batch/trunk/reference/html/metaDataSchema.html BATCH_STEP_EXECUTION_CONTEXT
- 17. Spring Batch Data Model http://docs.spring.io/spring-batch/trunk/reference/html/metaDataSchema.html BATCH_STEP_EXECUTION_CONTEXT BATCH_JOB_INSTANCE BATCH_JOB_EXECUTION BATCH_JOB_EXECUTION_PARAMS BATCH_STEP_EXECUTION
- 18. Main Benefits • Integrated with full Spring framework supporting all Spring features such as Dependency Injection, AOP & Testability. • Well tested ItemReaders and ItemWriters covering vast options for I/O (File, JDBC, NoSQL JMS). • Management information of running and finished Batch processes, including information about results. • A collection of Tasklets (Spring Batchlets) for execution from shell scripts to Hadoop processes. • Framework flexibility to define complex steps configurations. • Simplification of thread management and transactions management. • Management of Batch lifecycle (Restartability, Run, Pause, Stop).
- 19. SPRING BATCH AT A BANK
- 20. BANK • The bank is undergoing a process of migration of core applications from Host to a mid-level system that uses a Spring-based architecture defined, developed and maintained by everis. Time to market MIPS reduction Scalability TestabilityCost reduction HOST MID-LEVEL SYSTEMS
- 21. Architecture • Spring based architecture: – Performs 7.000 transactions per second. – Provides service to 30.000 users – Architecture team composed by 30 software engineers. INFR. CORPORATE CHANNEL SERVICES HOST LOGIN IDM BATCH BACKENDS ONLINE CHANNEL CRYPTO NEUTRAL CHANNEL OTHER MEDIA STORAGE FINANCIAL TERMINAL ONLINE BANKING GROUP COMPANIES ATM BATCH
- 22. Challenge HOST How to integrate Spring Batch in corporate tools? SPRINGBATCH
- 24. Integration Corporate batch launcher ClassPathXmlApplicationContext sbc = BatchApplicationContextUtils.createApplicationContext(properties); JobLauncher launcher = (JobLauncher) sbc.getBean("jobLauncher"); Job job = (Job) sbc.getBean(this.jobId); initBatchContext(jobParameters, appParams, jobParam, sbc); public static ClassPathXmlApplicationContext createApplicationContext(Properties credentials) throws IOException { String[] contextFiles = new String[]{ "classpath*:batch/arch/internal/beans/-batch-arch*.xml", "classpath*:batch/app/**/-batch-app*.xml" }; ClassPathXmlApplicationContext sbc = new ClassPathXmlApplicationContext(contextFiles, false); PropertyPlaceholderConfigurer ppc = new PropertyPlaceholderConfigurer(); ppc.setSystemPropertiesMode(PropertyPlaceholderConfigurer.SYSTEM_PROPERTIES_MODE_NEVER); ppc.setIgnoreUnresolvablePlaceholders(true); ppc.setProperties(credentials); sbc.addBeanFactoryPostProcessor(ppc); sbc.refresh(); return sbc; } protected void initBatchContext(JobParameters jobParams, String[] appParams, String[] jobParam, ClassPathXmlApplicationContext xmlcontext) { BatchContext.reset(); BatchContext bc = BatchContext.getInstance(); BatchParams bp = new BatchParams(); bp.setJobName(this.jobId); bp.setInstanceId(jobParams.getString(BatchLauncher.ArchParamKey.INSTANCIA.toString())); bp.setScheduleTimestamp(jobParams.getDate(BatchLauncher.ArchParamKey.SCHED_TS.toString())); bp.setAppParams(Arrays.asList(appParams)); bp.setJobParam(Arrays.asList(jobParam)); bp.setJobentorno(jobParams.getString(BatchLauncher.ArchParamKey.ENTORNO.toString())); bp.setJobappmvs(jobParams.getString(BatchLauncher.ArchParamKey.APPMVS.toString())); bc.setBatchParameters(bp); bc.setxmlContext(xmlcontext); }
- 25. Results Main areas with Batch processes: • Markets • IT • Risk Management • Digital Contents • Banking services • ATMs • Online Banking • Extranet 0 50000 100000 150000 200000 250000 ADCAEX AGERIE AMPAMP ARTBKD AUWATC BCKARM BCKPBC BCKSDD BKDAIR BKDGBR BKDPRP CALBCK COMCNF ECCVAL EPQEPQ GESINS IMPIMP LVACUA MUROGE PERCAI PUNBKD SELCLI SPCSPC TRACEA BSMTST IDECLD SISSEG # executions per process # executions per month
- 26. SPRING BATCH AT AN ENERGY COMPANY
- 27. ENERGY COMPANY • Software factory composed by 4 different providers. • More than 300 developers. • Migration from custom developed framework to a Spring based architecture.
- 29. Challenges 1. How to migrate current custom batch processes on new Spring operational architecture? 1. How to ensure quality and correct Spring Batch usage with a community of 300 developers 2. How to monitor and report batch activity formed by more than 400 processes with 160 processes per day executions?
- 30. 1. MIGRATION
- 31. Migration PoC Objectives: • Evaluate the effort required by the development teams to migrate. • Evaluate the performance. • Evaluate the total execution time. • Identify improvements over migrated process.
- 32. Migration • Obtained results: 30% Reduction of the Execution time 15% Reduction of the Memory consumption 10% Reduction of the CPU consumption Process migration, including tests, has been performed in 8 Working Days without previous Spring Batch experience. Reduction of the proprietary code. Maintainability on batch processes (Query + DAO + Processor + Config writer). • Where do we obtain performance benefits? – Using JDBCPagingItemReader, we eliminate a reading lock at the beginning of the original process. – Using ApacheCommonsDBCP as a Datasource implementation for Reader, we optimize the process as the old architecture component had performance issues. – Using FlatFileItemWriter to generate CSV output file, the results are improved compared to the ones previously obtained
- 33. 2. QUALITY
- 34. Quality
- 35. Quality LOCAL TEST ACCEPTANCE TRAINING PRODUCTION PRE PRODUCTION PERFORMANCE Productive environmentsTactic environments Development environments Monitoring QA environments Dynamic Code AnalysisReporting & Bugtracking Static code analysis Performance testing & profiling Continuous Integration Static Data Model Analysis OLTP
- 36. 3. MONITOR AND CONTROL
- 37. Monitor and control 01:00 AM 02:00 AM 03:00 AM Planed 04:00 AM 05:00 AM 06:00 AM 01:00 AM 02:00 AM 03:00 AM Happens 04:00 AM 05:00 AM 06:00 AM
- 43. Needs Monitor and Control Batch Executions • No global and complete reporting tool around Batch processes. • Difficulties on schedule modifications because huge number of processes (432) and high number of executions (160 per day). • No alert mechanism if planned Batch doesn’t start (Planned vs Executed). Batch Plannifications
- 44. Monitor and Control 44 Batch Executions Logs Planificación Batch • Daily global information. • Visual control of executions vs planification • Historical Data..
- 45. Conclusions • Structure Batch following standards and best practices. • Well tested and optimized components. • Possibilities to exploit horizontal scalability framework offer. • Integration with the rest of the Spring framework (Spring Integration / Spring XD). • Reusability of business logic developed in online applications. • Consider the complexity of integrating Spring Batch with existing enterprise tools. • Migration process from current batches must be normalized to SpringBatch characteristics. • Ensure invariability of the process after migration using appropriate testing tools. • Ensure correctness of framework usage using checkstyle tools. • Design batch having in mind advanced framework capacities (chunking, parallelism). • Measure processes execution ensuring correctness during all process life Benefits Advises
- 48. Spring Batch for large Enterprises operations Ignasi Gonzalez @ignasi15g