Batch processing
- 1. Mule Batch Processing Concepts
- 2. Batch Processing Mule possesses the ability to process messages in batches. Within an application, you can initiate a batch job which is a block of code that splits messages into individual records, performs actions upon each record, then reports on the results and potentially pushes the processed output to other systems or queues.
- 3. Batch processing is particularly useful when working with the following scenarios: Integrating data sets, small or large, streaming or not, to parallel process records. Synchronizing data sets between business applications, such as syncing contacts between NetSuite and Salesforce and causing "near real-time" data integration. Extracting, transforming and loading (ETL) information into a target system, such as uploading data from a flat file (CSV) to Hadoop. Handling large quantities of incoming data from an API into a legacy system.
- 4. Batch Job: Batch job is a top-level element in Mule which exists outside all Mule flows. Batch jobs split large messages into records which processes asynchronously in a batch job. A batch job contains one or more batch steps which, in turn, contain any number of message processors that act upon records as they move through the batch job. During batch processing, you can use record variables and MEL expressions to enrich, route or otherwise act upon records. Batch Job Insatnce: whenever a Mule flow executes a batch job. Mule creates the batch job instance in the Load and Dispatch phase. Every batch job instance is identified internally using a unique String known as batch job instance id.
- 5. Phase Configuration Input optional Load and Dispatch implicit, not exposed in a Mule application Process required On Complete optional Batch Processing Phases:
- 6. Input: The first phase, Input, is an optional part of the batch job configuration and is designed to Triggering Batch Jobs via an inbound connector, and/or accommodate any transformations or adjustments to a message payload before Mule begins processing it as a batch. Load and Dispatch: The second phase, Load and Dispatch, is implicit and performs all the "behind the scenes" work to create a batch job instance. Essentially, this is the phase during which Mule turns a serialized message payload into a collection of records for processing as a batch. You don’t need to configure anything for this activity to occur, though it is useful to understand the tasks Mule completes during this phase. Process: Mule begins asynchronous processing of the records in the batch. Within this required phase, each record moves through the message processors in the first batch step, then is sent back to the original queue while it waits to be processed by the second batch step and so on until every record has passed through every batch step.
- 7. Only one queue exists and records are picked out of it for each batch step, processed, and then sent back to it each record keeps track of what stages it has been processed through while it sits on this queue. Note that a batch job instance does not wait for all its queued records to finish processing in one batch step before pushing any of them to the next batch step.
- 8. On Complete: we can optionally configure Mule to create a report or summary of the records it processed for the particular batch job instance. This phase exists to give system administrators and developers some insight into which records failed so as to address any issues that might exist with the input data. After Mule executes the entire batch job, the output becomes a batch job result object (BatchJobResult). you have two options for working with the output. − Create a report − Reference the batch job result object
- 9. Batch Processing Terminology: we have term in batch processing. Batch,Batch Commit,Batch Job,Batch Job Instance,Batch Job Result,Batch Message Processor,Batch Phase,Batch Step,Record Batch Elements: − Batch,Batch Commit,Batch Reference,Batch Threading Profile, Record Variable. − In Batch Threading Profile, we have some attributes. PoolExhaustedAction,maxThreadsActive,maxThreadsIdle, threadTTL,threadWaitTimeout,maxBufferSize BatchJobResult Processing Statistics: batchJobInstanceId,elapsedTimeInMillis,failedOnCompletePhase,fail edOnInputPhase,failedOnLoadingPhase,failedRecords,inputPhase Exception,loadedRecords,loadingPhaseException,onCompletePha seException,processedRecords,successfulRecords,totalRecords.
- 10. Handling Failures During Batch Processing: Mule has three options for handling a record-level error. Finish processing Continue processing Continue processing based on limit