Spring Batch Behind the Scenes

Spring Batch Bootcamp

• Your host: Josh Long
SpringSource, a division of VMware

• Code: github.com/joshlong/spring-batch-bootcamp
• Deck: slideshare.net/joshlong/spring-batch-behind-the-scenes

1

ahuge amount of this came from Dr. Dave Syer
(he’s awesome)

2

About Josh Long (Spring Developer Advocate)

@starbuxman
josh.long@springsource.com
3

Agenda

• Introduce Spring Batch
– concepts
– speciﬁcs
– demos

5

Inside Spring Batch

• Architecture and Domain Overview
• Application concerns and Getting Started
• Chunk-Oriented Processing

6

Inside Spring Batch


7

Spring Batch: Layered Architecture

Application

Batch Core
Batch Execution
Environment

Infrastructure

8


Business Domain
– Record-level
data (e.g. Trade)

Application

Batch Core
Batch Execution Batch Domain -
Environment Job, Chunk, Step,
Partition, Status

Repeat, Retry,
Infrastructure Transaction,
Input/Output

8


Business Domain
– Record-level
data (e.g. Trade)

Application
Publicly exposed
Batch Execution
Environment
APIs
Batch Core
Batch Execution Batch Domain -
Environment Job, Chunk, Step,
Partition, Status

Repeat, Retry,
Infrastructure Transaction,
Input/Output

8

Spring Batch Dependencies

Spring Batch
Samples

Application Core

Execution Infrastructure

Spring Core
Compile
Configuration
Spring Framework

9

Batch Domain Diagram

Batch
JobParameters uses Operator
uses to identify and manage jobs

JobLauncher
uses to construct jobs
starts and stops
executes
JobInstance
recipe for
*

StepInstance

Job
stored in
*

Step

Database
Application configures
Developer

10

Job Conﬁguration and Execution

The EndOfDay Job
Job

* The EndOfDay Job
JobInstance for 2011/05/05

* The first attempt at
JobExecution EndOfDay Job
for 2011/05/05

11

Job and Step

Job
* Step
*
JobInstance

Step Scope
*
JobExecution
* StepExecution

12

DEMO of Spring Batch Application

13

Inside Spring Batch


14

Application Concerns

• Getting Started
• Stateful or Stateless
• Step Scope
• Domain Speciﬁc Language
• Resource Management
• Alternative Approaches
• Failure Modes

15


• Getting Started
• Step Scope
• Failure Modes

16

Getting Started

Application
Developer
implements ItemProcessor (optional)

input output (optional)
configures
ItemReader ItemWriter

Job

* StepExecutor concerns
Step

RepeatOperations ExceptionHandler

17

OK, So How Do I start?

• Find and install the appropriate .sql script in your database
– they live in org.springframework.batch.core in spring-batch-core.jar

18

@Inject JobLauncher launcher ;
@Inject @Qualifier("importData") Job job ;

@Schedule(cron = "* 15 9-17 * * MON-FRI ")
public void run15MinutesPastHourDuringBusinessDays() throws Throwable {

Resource samplesResource = new ClassPathResource("/sample/a.csv");
String absFilePath = "file:///" + samplesResource.getFile().getAbsolutePath();

JobParameters params = new JobParametersBuilder()
.addString("input.file", absFilePath)
.addDate("date", new Date())
.toJobParameters();

JobExecution jobExecution = jobLauncher.run(job, params);
BatchStatus batchStatus = jobExecution.getStatus();
while (batchStatus.isRunning()) Thread.sleep(1000);
JobInstance jobInstance = jobExecution.getJobInstance();
}
19


• Or... Deploy the Spring Batch Admin
– good for operations types
– good for auditing the batch jobs

20

DEMO of Spring Batch Admin

21

ItemReader

public interface ItemReader<T> {

T read() throws Exception,
UnexpectedInputException,
ParseException,
NonTransientResourceException;

}

Returns null at end of dataset

delegate Exception handling to framework

22

Database Cursor input

• Cursor is opened over all data that will be input for a given job
• Each row in the cursor is one ‘item’
• Each call to read() will advance the ResultSet by one row, and
return one item that is equivalent to one row

23

Database Cursor Input

ID NAME BAR
1 foo1 bar1
2 foo2 bar2
3 foo3 bar3
4 foo4 bar4
5 foo5 bar5
6 foo6 bar6
7 foo7 bar7
8 foo8 bar8

24


FOO 2 Select * from FOO
id=2 where id > 1 and id < 7
name=foo2
bar=bar2 ID NAME BAR
1 foo1 bar1
2 foo2 bar2
3 foo3 bar3
4 foo4 bar4
5 foo5 bar5
6 foo6 bar6
7 foo7 bar7
8 foo8 bar8

24


Select * from FOO
where id > 1 and id < 7

ID NAME BAR
1 foo1 bar1
FOO 3
2 foo2 bar2
id=3
name=foo3 3 foo3 bar3
bar=bar3 4 foo4 bar4
5 foo5 bar5
6 foo6 bar6
7 foo7 bar7
8 foo8 bar8

24


Select * from FOO
where id > 1 and id < 7

ID NAME BAR
1 foo1 bar1
2 foo2 bar2
3 foo3 bar3
4 foo4 bar4
5 foo5 bar5
FOO 4
6 foo6 bar6
id=4
name=foo4 7 foo7 bar7
bar=bar4 8 foo8 bar8

24

Database Cursor input

@Bean
public JdbcCursorItemReader reader (){
JdbcCursorItemReader reader = new JdbcCursorItemReader();
reader.setDataSource(dataSource());
reader.setVerifyCursorPosition(true);
reader.setRowMapper( new PlayerSummaryMapper());
reader.setSql("SELECT GAMES.player_id, GAMES.year_no, SUM(COMPLETES), "+
"SUM(ATTEMPTS), SUM(PASSING_YARDS), SUM(PASSING_TD), "+
"SUM(INTERCEPTIONS), SUM(RUSHES), SUM(RUSH_YARDS), "+
"SUM(RECEPTIONS), SUM(RECEPTIONS_YARDS), SUM(TOTAL_TD) "+
"from GAMES, PLAYERS where PLAYERS.player_id = "+
"GAMES.player_id group by GAMES.player_id, GAMES.year_no");
return reader;
}

25

Xml input

• Xml ﬁles are separated into fragments based on a root
element
• Each fragment is sent to Spring OXM for binding.
• One fragment is processed per call to read().
• Synchronized with the transaction to ensure any rollbacks
won’t cause duplicate records.

26

Xml Input

<trade>
<isin>XYZ0001</isin>
Fragment 1 <quantity>5</quantity>
<price>11.39</price>
<customer>Customer1</customer>
</trade>
<trade>
Fragment 2 <quantity>2</quantity>
<price>72.99</price>
<customer>Customer2c</customer>
</trade>
<trade>
<quantity>9</quantity>
Fragment 3 <price>99.99</price>
<customer>Customer3</customer>
</trade>

27

Xml Input

Spring OXM

JaxB2
Fragment 1
Fragment 2
Fragment 3
Castor

XmlBeans

Any binding framework
supported by Spring OXM

28

Flat File input

• How lines are read from a file is separated from how a line
is parsed allowing for easy support of additional file
formats.
• Parsing is abstracted away from users
• Familiar usage patterns: FieldSets can be worked with in
similar ways as ResultSets
• Supports both Delimited and Fixed-Length file formats.

29

LineTokenizer

public interface LineTokenizer {
FieldSet tokenize(String line);
}

FieldSet Line
Tokenizer UK21341EAH45,978,98.34,customer1
UK21341EAH46,112,18.12,customer2
UK21341EAH47,245,12.78,customer2
UK21341EAH48,108,109.25,customer3
UK21341EAH49,854,123.39,customer4

30

FieldSetMapper

public class TradeFieldSetMapper
implements FieldSetMapper<Trade> {

public Trade mapFieldSet(FieldSet fieldSet)
throws BindException {

Trade trade = new Trade();
trade.setIsin(fieldSet.readString(0));
trade.setQuantity(fieldSet.readLong(1));
trade.setPrice(fieldSet.readBigDecimal(2));
trade.setCustomer(fieldSet.readString(3));
return trade;
}
}

31

Column-name Access to FieldSet

@Bean
public DelimitedLineTokenizer tradeTokenizer() throws Exception {
DelimitedLineTokenizer dlt = new DelimitedLineTokenizer();
dlt.setDelimiter( DelimitedLineTokenizer.DELIMITER_COMMA);
dlt.setNames( "ISIN,Quantity,Price,Customer".split(","));
return dlt;
}

trade.setIsin(fieldSet.readString(0));
trade.setQuantity(fieldSet.readLong(1));
trade.setPrice(fieldSet.readBigDecimal(2));
trade.setCustomer(fieldSet.readString(3));
return trade;

32


@Bean
return dlt;
}

32


@Bean
return dlt;
}

trade.setIsin( fieldSet.readString(“ISIN”));
trade.setQuantity( fieldSet.readLong(“Quantity”));
trade.setPrice( fieldSet.readBigDecimal(“Price”));
trade.setCustomer( fieldSet.readString(“Customer”));
return trade;

32

Flat File Parsing errors

• It is extremely likely that bad data will be read in, when this
happens information about the error is crucial, such as:
– Current line number in the ﬁle
– Original Line
– Original Exception (e.g. NumberFormatException)
• Allows for detailed logs, that can be processed on an ad-
hoc basis, or using a speciﬁc job for bad input data.

33

ItemProcessor

public interface ItemProcessor<I, O> {
O process(I item) throws Exception;
}

Delegate Exception handling to framework

34

Item processors

• optional
– simple jobs may be constructed entirely with out-of-box
readers and writers
• sit between input and output
• typical customization site for application developers
• good place to coerce data into the right format for output
• chain transformations using CompositeItemProcessor

35

ItemWriter

public interface ItemWriter<T> {

void write(List<? extends T> items) throws Exception;

}

expects a “chunk”

delegate Exception handling to framework

36

Item Writers

• handles writing and serializing a row of data
• the input might be the output of a reader or a processor
• handles transactions if necessary and associated
rollbacks

37

ItemWriters

@Value("#{systemProperties['user.home']}")
private String userHome;

@Bean @Scope(“step”)
public FlatFileItemWriter writer( ){
FlatFileItemWriter w = new FlatFileItemWriter();
w.setName( "fw1");
File out = new File( this.userHome,
"/batches/results").getAbsolutePath();
Resource res = new FileSystemResource(out);
w.setResource(res);
return w;
}

38

ItemWriters
@Bean
public JpaItemWriter jpaWriter() {
JpaItemWriter writer = new JpaItemWriter();
writer.setEntityManagerFactory( entityManagerFactory() );
return writer;
}

39


• Getting Started
• Step Scope
• Failure Modes

40

Stateful or Stateless?

Job

JobInstance JobInstance

2 JobInstances can access the same
ItemWriter concurrently

BusinessItemWriter

41

Stateful or Stateless: StepContext

Job

JobInstance JobInstance

Concrete ItemWriters only created
once per step execution as needed

step scoped proxy

BusinessItemWriter Proxy BusinessItemWriter

Job has reference to Proxy

42

Introducing the Step scope

File writer needs to be step scoped so it can flush and close the output stream
@Scope("step")
@Bean
Make this bean injectable
public FlatFileItemReader reader(
@Value("#{jobParameters['input.ﬁle']}")
Resource input ){
FlatFileItemReader fr = new FlatFileItemReader();
fr.setResource(input);
fr.setLineMapper( lineMapper() ); Inner beans inherit the
fr.setSaveState(true); enclosing scope by default
return fr;
}

Because it is step scoped the bean has access to the
StepContext values and can get at it through Spring EL (in Spring >3)

43

Step Scope Responsibilities

• Create beans for the duration of a step
• Respect Spring bean lifecycle metadata (e.g.
InitializingBean at start of step, DisposableBean at end
of step)
• Allows stateful components in a multithreaded
environment

44


• Getting Started
• Step Scope
• Failure Modes

45

Domain Specific Language

• Keeping track of all the application concerns can be a
large overhead on a project
• Need a DSL to simplify configuration of jobs
• DSL can hide details and be aware of specific things like
well-known infrastructure that needs to be step scoped
• The Spring way of dealing with this is to use a custom
namespace

46

XML Namespace Example
<job id="skipJob" incrementer="incrementer"
xmlns="http://www.springframework.org/schema/batch"> chunk has input,
output and a
processor
<step id="step1">
<tasklet>
<chunk reader="ﬁleItemReader" processor="tradeProcessor"
writer="tradeWriter" commit-interval="3" skip-limit="10">
<skippable-exception-classes> lots of control over
<include class="....FlatFileParseException" /> errors
<include class="....WriteFailedException" />
</skippable-exception-classes>
</chunk> flow control from one
</tasklet> step to another
<next on="*" to="step2" />
<next on="COMPLETED WITH SKIPS" to="errorPrint1" />
<fail on="FAILED" exit-code="FAILED" />
</step>

...

</job>
47


• Getting Started
• Step Scope
• Failure Modes

48

Resource Management Responsibilities

• Open resource (lazy initialisation)
• Close resource at end of step
• Close resource on failure
• Synchronize output file with transaction – rollback resets
file pointer
• Synchronize cumulative state for restart
– File pointer, cursor position, processed keys, etc.
– Statistics: number of items processed, etc.
• Other special cases
– Hibernate flush policy
– JDBC batch update

49

File Output Resource Synchronization

TransactionSynchronizationManager

Writer TransactionSynchronization FileResource

write(items)

register(…) create
mark()

write(items)

write(items)

50




write(items)

mark()

write(items)

write(items)

Error!

50




write(items)

mark()

write(items)

write(items)

Error!
rollback()
reset()

50

Hibernate Flush Policy

• First problem
– If we do not flush manually, transaction manager handles
automatically
– …but exception comes from inside transaction manager, so
cannot be caught and analyzed naturally by StepExecutor
– Solution: flush manually on chunk boundaries
• Second problem
– Errors cannot be associated with individual item
– Two alternatives
• Binary search through chunk looking for failure = O(logN)
• Aggressively flush after each item = O(N)
• HibernateItemWriter

51


• Getting Started
• Step Scope
• Failure Modes

52

Tasklet: Alternative Approaches for Application Developer

• Sometimes Input/Output is not the way that a job is
structured
• E.g. Stored Procedure does everything in one go, but we
want to manage it as a Step in a Job
• E.g. repetitive process where legacy code prevents
ItemReader/Processor being identiﬁed
• For these cases we provide Tasklet

53

Tasklet
public interface Tasklet {

RepeatStatus execute(StepContribution contribution,
ChunkContext chunkContext) throws Exception;
}

Signal to framework about end of business process:

RepeatStatus.CONTINUABLE delegate Exception handling to framework
RepeatStatus.FINISHED

54

Getting Started With a Tasklet

Application
Developer
implements Tasklet

configures

Job

* StepExecutor concerns
Step

RepeatOperations ExceptionHandler

55

XML Namespace Example with Tasklet

Tasklet Step has
Tasklet

<job id="loopJob" xmlns="http://www.springframework.org/schema/batch">
<step id="step1">
<tasklet ref="myCustomTasklet">
<transaction-attributes propagation="REQUIRED"/>
</tasklet>
</step>
</job>

56


• Getting Started
• Step Scope
• Failure Modes

57

Failure Modes
Event Response Alternatives

Bad input record (e.g. Mark item as skipped but Abort Job when skipLimit
parse exception) exclude from Chunk reached

Bad output record – Mark item as skipped. Abort Job when skipLimit
business or data integrity Retry chunk excluding reached (after Chunk
exception bad item. completes)

Bad output record – Retry Chunk including Abort Chunk when retry
deadlock loser exception bad item. limit reached

Bad Chunk – e.g. Jdbc Retry Chunk but flush Discard entire chunk;
batch update fails and commit after every binary search for failed
item item

Output resource failure – Graceful abort. Attempt If database is unavailable
e.g. disk full, permanent to save batch data. meta data remains in
network outage “running” state!

58

Inside Spring Batch


59

Chunk-Oriented Processing

• Input-output can be grouped together = Item-Oriented
Processing (e.g. Tasklet)
• …or input can be aggregated into chunks ﬁrst = Chunk-
Oriented Processing (the chunk element of a tasklet)
• Chunk processing can be encapsulated, and independent
decisions can be made about (e.g.) partial failure and retry
• Step = Chunk Oriented (default, and more common)
• Tasklet = Item Oriented
• Here we compare and contrast the two approaches

60

Item-Oriented Pseudo Code

REPEAT(while more input) {

TX {
REPEAT(size=500) {

input;
output;

}
}

}

61

Item-Oriented Pseudo Code

RepeatTemplate

TX { TransactionTemplate
REPEAT(size=500) {
RepeatTemplate
input;
output;

}
} Business Logic

}

61

Item-Oriented Retry and Repeat

REPEAT(while more input
AND exception[not critical]) {
TX {
REPEAT(size=500) {
RETRY(exception=[deadlock loser]) {
input;
} PROCESS {
output;
} SKIP and RECOVER {
notify;
}
}
}
}

62

Chunk-Oriented Pseudo Code

chunk = ACCUMULATE(size=500) {
input;
}
RETRY {
TX {
for (item : chunk) { output; }
}
}
}

63

Summary

• Spring Batch manages the loose ends so you can sleep
• Easy to Use Along with Other Spring Services
• Things We Didn’t talk about:
– remote chunking
– all the ItemReaders/Writers
• Spring Batch Admin Lets Operations Types Sleep Easier

64

Spring Batch Behind the Scenes

More Related Content

What's hot (20)

Viewers also liked (12)

Similar to Spring Batch Behind the Scenes (20)

More from Joshua Long (20)

Recently uploaded (20)

Spring Batch Behind the Scenes

Editor's Notes