Process CSV file using Batch Job

 

Before starting with the POC we will strongly recommend to go through our Batch Processing tutorial to have a knowledge around how Batch Job work. Let’s see how we can process CSV file using Batch Job processing.

For the POC

  • We will process CSV having only 9 records – Sample CSV
    • Total 9 data records
    • 7 are valid records
    • 2 records will fail – having region as Europe (to show failure scenario)
  • Batch Job contain following Batch Steps
    • Batch_Step_Process – To process the records as per business needs
      • Batch Aggregator_DB_Insert – to bulk insert records into DB in batch of 5 records
    • Batch_Step_Failed_Records – Step to process failed records
      • Batch Aggregator Send Email – to send email containing all failed records

Sample.csv

Region,Country,Item Type,Sales Channel,Order Priority,Order Date,Order ID,Ship Date,Units Sold,Unit Price,Unit Cost,Total Revenue,Total Cost,Total Profit
Australia and Oceania,Tuvalu,Baby Food,Offline,H,5/28/2010,669165933,6/27/2010,9925,255.28,159.42,2533654.00,1582243.50,951410.50
Central America and the Caribbean,Grenada,Cereal,Online,C,8/22/2012,963881480,9/15/2012,2804,205.70,117.11,576782.80,328376.44,248406.36
Europe,Russia,Office Supplies,Offline,L,5/2/2014,341417157,5/8/2014,1779,651.21,524.96,1158502.59,933903.84,224598.75
Sub-Saharan Africa,Sao Tome and Principe,Fruits,Online,C,6/20/2014,514321792,7/5/2014,8102,9.33,6.92,75591.66,56065.84,19525.82
Sub-Saharan Africa,Rwanda,Office Supplies,Offline,L,2/1/2013,115456712,2/6/2013,5062,651.21,524.96,3296425.02,2657347.52,639077.50
Australia and Oceania,Solomon Islands,Baby Food,Online,C,2/4/2015,547995746,2/21/2015,2974,255.28,159.42,759202.72,474115.08,285087.64
Europe,Angola,Household,Offline,M,4/23/2011,135425221,4/27/2011,4187,668.27,502.54,2798046.49,2104134.98,693911.51
Sub-Saharan Africa,Burkina Faso,Vegetables,Online,H,7/17/2012,871543967,7/27/2012,8082,154.06,90.93,1245112.92,734896.26,510216.66
Sub-Saharan Africa,Republic of the Congo,Personal Care,Offline,M,7/14/2015,770463311,8/25/2015,6070,81.73,56.67,496101.10,343986.90,152114.20

 

Create a mule application and drag and drop On New or Updated File from File connector – this will monitor the folder for any new or updated file

Process CSV file using Batch Job

 

Add the File Config

Process CSV file using Batch Job

 

Drag and drop Batch Job from Core activities

Process CSV file using Batch Job

 

Batch Job will have following configuration

  • Will set the Max Failed records to -1 as we want to process whole CSV file

 

Now we will add the batch step to do following processing

  • Accept All or No failures records as accept policy
  • If Region in CSV has a value of Europe then we will raise a custom exception – added choice
  • Otherwise log the message – processing #[payload]
  • Add Batch Aggregator
    • Set the aggregator size to 5
    • Transform the payload collection of 5 records to DB insert request
    • Log the message – insert into DB #[payload]

So the updated batch step will look below

 

Now we will add another batch step to handle all the failure records

  • Accept Only Failures records as accept policy
  • Logs the message – failed record found #[payload]
  • Add Batch Aggregator
    • Set the aggregator size to Streaming (means for all the failed records)
    • Transform the payload to email payload
    • Log the message – sending email for records #[payload]

So the batch step for processing failed records will look like below

 

And in last will add log in On Complete phase

  • Log message – summary report – #[payload write “application/json”]

 

Now if we run the application for Sample CSV we can see below logs

If we do the analysis of the logs we can see full logs and the analysis

Full batch processing log

INFO  2020-06-04 16:54:59,886 [[MuleRuntime].io.05: [batch-job-sample].batch-job-sampleFlow.BLOCKING @7a774c84] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] com.mulesoft.mule.runtime.module.batch.internal.engine.DefaultBatchEngine: Created instance '06bf73b0-a656-11ea-a5d6-9aaf65ed66d8' for batch job 'batch-job-sample_Batch_Job'
INFO  2020-06-04 16:54:59,887 [[MuleRuntime].io.05: [batch-job-sample].batch-job-sampleFlow.BLOCKING @7a774c84] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] com.mulesoft.mule.runtime.module.batch.internal.engine.queue.BatchQueueLoader: Starting loading phase for instance '06bf73b0-a656-11ea-a5d6-9aaf65ed66d8' of job 'batch-job-sample_Batch_Job'
INFO  2020-06-04 16:54:59,907 [[MuleRuntime].io.05: [batch-job-sample].batch-job-sampleFlow.BLOCKING @7a774c84] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] com.mulesoft.mule.runtime.module.batch.internal.engine.queue.BatchQueueLoader: Finished loading phase for instance 06bf73b0-a656-11ea-a5d6-9aaf65ed66d8 of job batch-job-sample_Batch_Job. 9 records were loaded
INFO  2020-06-04 16:54:59,921 [[MuleRuntime].io.05: [batch-job-sample].batch-job-sampleFlow.BLOCKING @7a774c84] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] com.mulesoft.mule.runtime.module.batch.internal.engine.DefaultBatchEngine: Started execution of instance '06bf73b0-a656-11ea-a5d6-9aaf65ed66d8' of job 'batch-job-sample_Batch_Job'
INFO  2020-06-04 16:54:59,937 [[MuleRuntime].io.07: [batch-job-sample].batch-job-batch-job-sample_Batch_Job-work-manager @72b2fd47] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: processing Region,Country,Item Type,Sales Channel,Order Priority,Order Date,Order ID,Ship Date,Units Sold,Unit Price,Unit Cost,Total Revenue,Total Cost,Total Profit
Australia and Oceania,Tuvalu,Baby Food,Offline,H,5/28/2010,669165933,6/27/2010,9925,255.28,159.42,2533654.00,1582243.50,951410.50

INFO  2020-06-04 16:54:59,941 [[MuleRuntime].io.07: [batch-job-sample].batch-job-batch-job-sample_Batch_Job-work-manager @72b2fd47] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: processing Region,Country,Item Type,Sales Channel,Order Priority,Order Date,Order ID,Ship Date,Units Sold,Unit Price,Unit Cost,Total Revenue,Total Cost,Total Profit
Central America and the Caribbean,Grenada,Cereal,Online,C,8/22/2012,963881480,9/15/2012,2804,205.70,117.11,576782.80,328376.44,248406.36

INFO  2020-06-04 16:54:59,944 [[MuleRuntime].io.07: [batch-job-sample].batch-job-batch-job-sample_Batch_Job-work-manager @72b2fd47] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] com.mulesoft.mule.runtime.module.batch.internal.DefaultBatchStep: Found exception processing record on step 'Batch_Step_Process' for job instance '06bf73b0-a656-11ea-a5d6-9aaf65ed66d8' of job 'batch-job-sample_Batch_Job'.
This is the first record to show this exception on this step for this job instance. Subsequent records with the same failureswill not be logged for performance and log readability reasons:

********************************************************************************
Message               : Raising error in case region is Europe
Error type            : MY:EXCEPTION
Element               : batch-job-sampleFlow/processors/0/route/0/route/0/processors/0/route/0/processors/0 @ batch-job-sample:batch-job-sample.xml:36 (Raise error)
Element XML           : <raise-error doc:name="Raise error" doc:id="4e4e2f36-4a42-479f-bc13-d295b3d5c8ea" type="MY:EXCEPTION" description="Raising error in case region is Europe"></raise-error>

  (set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
********************************************************************************

INFO  2020-06-04 16:54:59,948 [[MuleRuntime].io.07: [batch-job-sample].batch-job-batch-job-sample_Batch_Job-work-manager @72b2fd47] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: processing Region,Country,Item Type,Sales Channel,Order Priority,Order Date,Order ID,Ship Date,Units Sold,Unit Price,Unit Cost,Total Revenue,Total Cost,Total Profit
Sub-Saharan Africa,Sao Tome and Principe,Fruits,Online,C,6/20/2014,514321792,7/5/2014,8102,9.33,6.92,75591.66,56065.84,19525.82

INFO  2020-06-04 16:54:59,951 [[MuleRuntime].io.07: [batch-job-sample].batch-job-batch-job-sample_Batch_Job-work-manager @72b2fd47] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: processing Region,Country,Item Type,Sales Channel,Order Priority,Order Date,Order ID,Ship Date,Units Sold,Unit Price,Unit Cost,Total Revenue,Total Cost,Total Profit
Sub-Saharan Africa,Rwanda,Office Supplies,Offline,L,2/1/2013,115456712,2/6/2013,5062,651.21,524.96,3296425.02,2657347.52,639077.50

INFO  2020-06-04 16:54:59,955 [[MuleRuntime].io.07: [batch-job-sample].batch-job-batch-job-sample_Batch_Job-work-manager @72b2fd47] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: processing Region,Country,Item Type,Sales Channel,Order Priority,Order Date,Order ID,Ship Date,Units Sold,Unit Price,Unit Cost,Total Revenue,Total Cost,Total Profit
Australia and Oceania,Solomon Islands,Baby Food,Online,C,2/4/2015,547995746,2/21/2015,2974,255.28,159.42,759202.72,474115.08,285087.64

INFO  2020-06-04 16:54:59,958 [[MuleRuntime].cpuIntensive.05: [batch-job-sample].batch-job-sampleFlow.CPU_INTENSIVE @59dea51b] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: insert into DB  [
  "Region,Country,Item Type,Sales Channel,Order Priority,Order Date,Order ID,Ship Date,Units Sold,Unit Price,Unit Cost,Total Revenue,Total Cost,Total Profit\r\nAustralia and Oceania,Tuvalu,Baby Food,Offline,H,5/28/2010,669165933,6/27/2010,9925,255.28,159.42,2533654.00,1582243.50,951410.50\r\n",
  "Region,Country,Item Type,Sales Channel,Order Priority,Order Date,Order ID,Ship Date,Units Sold,Unit Price,Unit Cost,Total Revenue,Total Cost,Total Profit\r\nCentral America and the Caribbean,Grenada,Cereal,Online,C,8/22/2012,963881480,9/15/2012,2804,205.70,117.11,576782.80,328376.44,248406.36\r\n",
  "Region,Country,Item Type,Sales Channel,Order Priority,Order Date,Order ID,Ship Date,Units Sold,Unit Price,Unit Cost,Total Revenue,Total Cost,Total Profit\r\nSub-Saharan Africa,Sao Tome and Principe,Fruits,Online,C,6/20/2014,514321792,7/5/2014,8102,9.33,6.92,75591.66,56065.84,19525.82\r\n",
  "Region,Country,Item Type,Sales Channel,Order Priority,Order Date,Order ID,Ship Date,Units Sold,Unit Price,Unit Cost,Total Revenue,Total Cost,Total Profit\r\nSub-Saharan Africa,Rwanda,Office Supplies,Offline,L,2/1/2013,115456712,2/6/2013,5062,651.21,524.96,3296425.02,2657347.52,639077.50\r\n",
  "Region,Country,Item Type,Sales Channel,Order Priority,Order Date,Order ID,Ship Date,Units Sold,Unit Price,Unit Cost,Total Revenue,Total Cost,Total Profit\r\nAustralia and Oceania,Solomon Islands,Baby Food,Online,C,2/4/2015,547995746,2/21/2015,2974,255.28,159.42,759202.72,474115.08,285087.64\r\n"
]
INFO  2020-06-04 16:54:59,983 [[MuleRuntime].io.07: [batch-job-sample].batch-job-batch-job-sample_Batch_Job-work-manager @72b2fd47] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: processing Region,Country,Item Type,Sales Channel,Order Priority,Order Date,Order ID,Ship Date,Units Sold,Unit Price,Unit Cost,Total Revenue,Total Cost,Total Profit
Sub-Saharan Africa,Burkina Faso,Vegetables,Online,H,7/17/2012,871543967,7/27/2012,8082,154.06,90.93,1245112.92,734896.26,510216.66

INFO  2020-06-04 16:54:59,987 [[MuleRuntime].io.07: [batch-job-sample].batch-job-batch-job-sample_Batch_Job-work-manager @72b2fd47] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: processing Region,Country,Item Type,Sales Channel,Order Priority,Order Date,Order ID,Ship Date,Units Sold,Unit Price,Unit Cost,Total Revenue,Total Cost,Total Profit
Sub-Saharan Africa,Republic of the Congo,Personal Care,Offline,M,7/14/2015,770463311,8/25/2015,6070,81.73,56.67,496101.10,343986.90,152114.20

INFO  2020-06-04 16:55:00,041 [[MuleRuntime].io.07: [batch-job-sample].batch-job-batch-job-sample_Batch_Job-work-manager @72b2fd47] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] com.mulesoft.mule.runtime.module.batch.internal.DefaultBatchStep: Step Batch_Step_Process finished processing all records for instance 06bf73b0-a656-11ea-a5d6-9aaf65ed66d8 of job batch-job-sample_Batch_Job
INFO  2020-06-04 16:55:00,045 [[MuleRuntime].io.08: [batch-job-sample].batch-job-batch-job-sample_Batch_Job-work-manager @72b2fd47] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: failed record found Region,Country,Item Type,Sales Channel,Order Priority,Order Date,Order ID,Ship Date,Units Sold,Unit Price,Unit Cost,Total Revenue,Total Cost,Total Profit
Europe,Russia,Office Supplies,Offline,L,5/2/2014,341417157,5/8/2014,1779,651.21,524.96,1158502.59,933903.84,224598.75

INFO  2020-06-04 16:55:00,045 [[MuleRuntime].cpuIntensive.04: [batch-job-sample].batch-job-sampleFlow.CPU_INTENSIVE @59dea51b] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: insert into DB  [
  "Region,Country,Item Type,Sales Channel,Order Priority,Order Date,Order ID,Ship Date,Units Sold,Unit Price,Unit Cost,Total Revenue,Total Cost,Total Profit\r\nSub-Saharan Africa,Burkina Faso,Vegetables,Online,H,7/17/2012,871543967,7/27/2012,8082,154.06,90.93,1245112.92,734896.26,510216.66\r\n",
  "Region,Country,Item Type,Sales Channel,Order Priority,Order Date,Order ID,Ship Date,Units Sold,Unit Price,Unit Cost,Total Revenue,Total Cost,Total Profit\r\nSub-Saharan Africa,Republic of the Congo,Personal Care,Offline,M,7/14/2015,770463311,8/25/2015,6070,81.73,56.67,496101.10,343986.90,152114.20\r\n"
]
INFO  2020-06-04 16:55:00,047 [[MuleRuntime].io.08: [batch-job-sample].batch-job-batch-job-sample_Batch_Job-work-manager @72b2fd47] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: failed record found Region,Country,Item Type,Sales Channel,Order Priority,Order Date,Order ID,Ship Date,Units Sold,Unit Price,Unit Cost,Total Revenue,Total Cost,Total Profit
Europe,Angola,Household,Offline,M,4/23/2011,135425221,4/27/2011,4187,668.27,502.54,2798046.49,2104134.98,693911.51

INFO  2020-06-04 16:55:00,157 [[MuleRuntime].io.07: [batch-job-sample].batch-job-batch-job-sample_Batch_Job-work-manager @72b2fd47] [event: ] com.mulesoft.mule.runtime.module.batch.internal.DefaultBatchStep: Step Batch_Step_Failed_Records finished processing all records for instance 06bf73b0-a656-11ea-a5d6-9aaf65ed66d8 of job batch-job-sample_Batch_Job
INFO  2020-06-04 16:55:01,174 [[MuleRuntime].cpuIntensive.06: [batch-job-sample].batch-job-sampleFlow.CPU_INTENSIVE @59dea51b] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: sending email for records [
  [
    {
      "Region": "Europe",
      "Country": "Russia",
      "Item Type": "Office Supplies",
      "Sales Channel": "Offline",
      "Order Priority": "L",
      "Order Date": "5/2/2014",
      "Order ID": "341417157",
      "Ship Date": "5/8/2014",
      "Units Sold": "1779",
      "Unit Price": "651.21",
      "Unit Cost": "524.96",
      "Total Revenue": "1158502.59",
      "Total Cost": "933903.84",
      "Total Profit": "224598.75"
    }
  ],
  [
    {
      "Region": "Europe",
      "Country": "Angola",
      "Item Type": "Household",
      "Sales Channel": "Offline",
      "Order Priority": "M",
      "Order Date": "4/23/2011",
      "Order ID": "135425221",
      "Ship Date": "4/27/2011",
      "Units Sold": "4187",
      "Unit Price": "668.27",
      "Unit Cost": "502.54",
      "Total Revenue": "2798046.49",
      "Total Cost": "2104134.98",
      "Total Profit": "693911.51"
    }
  ]
]
INFO  2020-06-04 16:55:02,233 [[MuleRuntime].io.08: [batch-job-sample].batch-management-work-manager @71b08f1d] [event: ] com.mulesoft.mule.runtime.module.batch.internal.engine.DefaultBatchEngine: Starting execution of onComplete phase for instance 06bf73b0-a656-11ea-a5d6-9aaf65ed66d8 of job batch-job-sample_Batch_Job
INFO  2020-06-04 16:55:02,236 [[MuleRuntime].io.08: [batch-job-sample].batch-management-work-manager @71b08f1d] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: summary report - "{\n  \"onCompletePhaseException\": null,\n  \"loadingPhaseException\": null,\n  \"totalRecords\": 9,\n  \"elapsedTimeInMillis\": 1318,\n  \"failedOnCompletePhase\": false,\n  \"failedRecords\": 2,\n  \"loadedRecords\": 9,\n  \"failedOnInputPhase\": false,\n  \"successfulRecords\": 7,\n  \"inputPhaseException\": null,\n  \"processedRecords\": 9,\n  \"failedOnLoadingPhase\": false,\n  \"batchJobInstanceId\": \"06bf73b0-a656-11ea-a5d6-9aaf65ed66d8\"\n}"
INFO  2020-06-04 16:55:02,236 [[MuleRuntime].io.08: [batch-job-sample].batch-management-work-manager @71b08f1d] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] com.mulesoft.mule.runtime.module.batch.internal.engine.DefaultBatchEngine: Finished execution of onComplete phase for instance 06bf73b0-a656-11ea-a5d6-9aaf65ed66d8 of job batch-job-sample_Batch_Job
INFO  2020-06-04 16:55:02,236 [[MuleRuntime].io.08: [batch-job-sample].batch-management-work-manager @71b08f1d] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] com.mulesoft.mule.runtime.module.batch.internal.engine.DefaultBatchEngine: Finished execution for instance '06bf73b0-a656-11ea-a5d6-9aaf65ed66d8' of job 'batch-job-sample_Batch_Job'. Total Records processed: 9. Successful records: 7. Failed Records: 2
INFO  2020-06-04 16:55:02,237 [[MuleRuntime].io.08: [batch-job-sample].batch-management-work-manager @71b08f1d] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] com.mulesoft.mule.runtime.module.batch.internal.engine.DefaultBatchEngine: 
****************************************************************************************************************************************************************
*             - - + Exception Type + - -             *                  - - + Step + - -                  *                 - - + Count + - -                  *
****************************************************************************************************************************************************************
* com.mulesoft.mule.runtime.module.batch.exception.B * Batch_Step_Process                                 * 2                                                  *
****************************************************************************************************************************************************************

INFO  2020-06-04 16:55:02,260 [[MuleRuntime].io.08: [batch-job-sample].batch-management-work-manager @71b08f1d] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] com.mulesoft.mule.runtime.module.batch.internal.engine.DefaultBatchEngine: Finished execution for instance '06bf73b0-a656-11ea-a5d6-9aaf65ed66d8' of job 'batch-job-sample_Batch_Job'. Total Records processed: 9. Successful records: 7. Failed Records: 2

 

This is initial logs which are printed by the Load and Dispatch phase

INFO  2020-06-04 16:54:59,886 [[MuleRuntime].io.05: [batch-job-sample].batch-job-sampleFlow.BLOCKING @7a774c84] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] com.mulesoft.mule.runtime.module.batch.internal.engine.DefaultBatchEngine: Created instance '06bf73b0-a656-11ea-a5d6-9aaf65ed66d8' for batch job 'batch-job-sample_Batch_Job'
INFO  2020-06-04 16:54:59,887 [[MuleRuntime].io.05: [batch-job-sample].batch-job-sampleFlow.BLOCKING @7a774c84] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] com.mulesoft.mule.runtime.module.batch.internal.engine.queue.BatchQueueLoader: Starting loading phase for instance '06bf73b0-a656-11ea-a5d6-9aaf65ed66d8' of job 'batch-job-sample_Batch_Job'
INFO  2020-06-04 16:54:59,907 [[MuleRuntime].io.05: [batch-job-sample].batch-job-sampleFlow.BLOCKING @7a774c84] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] com.mulesoft.mule.runtime.module.batch.internal.engine.queue.BatchQueueLoader: Finished loading phase for instance 06bf73b0-a656-11ea-a5d6-9aaf65ed66d8 of job batch-job-sample_Batch_Job. 9 records were loaded
INFO  2020-06-04 16:54:59,921 [[MuleRuntime].io.05: [batch-job-sample].batch-job-sampleFlow.BLOCKING @7a774c84] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] com.mulesoft.mule.runtime.module.batch.internal.engine.DefaultBatchEngine: Started execution of instance '06bf73b0-a656-11ea-a5d6-9aaf65ed66d8' of job 'batch-job-sample_Batch_Job'

 

Process the first two correct records

INFO  2020-06-04 16:54:59,937 [[MuleRuntime].io.07: [batch-job-sample].batch-job-batch-job-sample_Batch_Job-work-manager @72b2fd47] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: processing Region,Country,Item Type,Sales Channel,Order Priority,Order Date,Order ID,Ship Date,Units Sold,Unit Price,Unit Cost,Total Revenue,Total Cost,Total Profit
Australia and Oceania,Tuvalu,Baby Food,Offline,H,5/28/2010,669165933,6/27/2010,9925,255.28,159.42,2533654.00,1582243.50,951410.50

INFO  2020-06-04 16:54:59,941 [[MuleRuntime].io.07: [batch-job-sample].batch-job-batch-job-sample_Batch_Job-work-manager @72b2fd47] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: processing Region,Country,Item Type,Sales Channel,Order Priority,Order Date,Order ID,Ship Date,Units Sold,Unit Price,Unit Cost,Total Revenue,Total Cost,Total Profit
Central America and the Caribbean,Grenada,Cereal,Online,C,8/22/2012,963881480,9/15/2012,2804,205.70,117.11,576782.80,328376.44,248406.36

 

For the Third record we can see the raise exception

Also we can see additional info – This is the first record to show this exception on this step for this job instance. Subsequent records with the same failureswill not be logged for performance and log readability reasons.

This is done to not hit the performance of batch processing for same set of errors

INFO  2020-06-04 16:54:59,944 [[MuleRuntime].io.07: [batch-job-sample].batch-job-batch-job-sample_Batch_Job-work-manager @72b2fd47] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] com.mulesoft.mule.runtime.module.batch.internal.DefaultBatchStep: Found exception processing record on step 'Batch_Step_Process' for job instance '06bf73b0-a656-11ea-a5d6-9aaf65ed66d8' of job 'batch-job-sample_Batch_Job'.
This is the first record to show this exception on this step for this job instance. Subsequent records with the same failureswill not be logged for performance and log readability reasons:

********************************************************************************
Message               : Raising error in case region is Europe
Error type            : MY:EXCEPTION
Element               : batch-job-sampleFlow/processors/0/route/0/route/0/processors/0/route/0/processors/0 @ batch-job-sample:batch-job-sample.xml:36 (Raise error)
Element XML           : <raise-error doc:name="Raise error" doc:id="4e4e2f36-4a42-479f-bc13-d295b3d5c8ea" type="MY:EXCEPTION" description="Raising error in case region is Europe"></raise-error>

  (set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
********************************************************************************

 

3 correct records

INFO  2020-06-04 16:54:59,948 [[MuleRuntime].io.07: [batch-job-sample].batch-job-batch-job-sample_Batch_Job-work-manager @72b2fd47] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: processing Region,Country,Item Type,Sales Channel,Order Priority,Order Date,Order ID,Ship Date,Units Sold,Unit Price,Unit Cost,Total Revenue,Total Cost,Total Profit
Sub-Saharan Africa,Sao Tome and Principe,Fruits,Online,C,6/20/2014,514321792,7/5/2014,8102,9.33,6.92,75591.66,56065.84,19525.82

INFO  2020-06-04 16:54:59,951 [[MuleRuntime].io.07: [batch-job-sample].batch-job-batch-job-sample_Batch_Job-work-manager @72b2fd47] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: processing Region,Country,Item Type,Sales Channel,Order Priority,Order Date,Order ID,Ship Date,Units Sold,Unit Price,Unit Cost,Total Revenue,Total Cost,Total Profit
Sub-Saharan Africa,Rwanda,Office Supplies,Offline,L,2/1/2013,115456712,2/6/2013,5062,651.21,524.96,3296425.02,2657347.52,639077.50

INFO  2020-06-04 16:54:59,955 [[MuleRuntime].io.07: [batch-job-sample].batch-job-batch-job-sample_Batch_Job-work-manager @72b2fd47] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: processing Region,Country,Item Type,Sales Channel,Order Priority,Order Date,Order ID,Ship Date,Units Sold,Unit Price,Unit Cost,Total Revenue,Total Cost,Total Profit
Australia and Oceania,Solomon Islands,Baby Food,Online,C,2/4/2015,547995746,2/21/2015,2974,255.28,159.42,759202.72,474115.08,285087.64

 

Batch Aggregator for DB insert – we can see 5 records as we defined the aggregator size as 5

INFO  2020-06-04 16:54:59,958 [[MuleRuntime].cpuIntensive.05: [batch-job-sample].batch-job-sampleFlow.CPU_INTENSIVE @59dea51b] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: insert into DB  [
  "Region,Country,Item Type,Sales Channel,Order Priority,Order Date,Order ID,Ship Date,Units Sold,Unit Price,Unit Cost,Total Revenue,Total Cost,Total Profit\r\nAustralia and Oceania,Tuvalu,Baby Food,Offline,H,5/28/2010,669165933,6/27/2010,9925,255.28,159.42,2533654.00,1582243.50,951410.50\r\n",
  "Region,Country,Item Type,Sales Channel,Order Priority,Order Date,Order ID,Ship Date,Units Sold,Unit Price,Unit Cost,Total Revenue,Total Cost,Total Profit\r\nCentral America and the Caribbean,Grenada,Cereal,Online,C,8/22/2012,963881480,9/15/2012,2804,205.70,117.11,576782.80,328376.44,248406.36\r\n",
  "Region,Country,Item Type,Sales Channel,Order Priority,Order Date,Order ID,Ship Date,Units Sold,Unit Price,Unit Cost,Total Revenue,Total Cost,Total Profit\r\nSub-Saharan Africa,Sao Tome and Principe,Fruits,Online,C,6/20/2014,514321792,7/5/2014,8102,9.33,6.92,75591.66,56065.84,19525.82\r\n",
  "Region,Country,Item Type,Sales Channel,Order Priority,Order Date,Order ID,Ship Date,Units Sold,Unit Price,Unit Cost,Total Revenue,Total Cost,Total Profit\r\nSub-Saharan Africa,Rwanda,Office Supplies,Offline,L,2/1/2013,115456712,2/6/2013,5062,651.21,524.96,3296425.02,2657347.52,639077.50\r\n",
  "Region,Country,Item Type,Sales Channel,Order Priority,Order Date,Order ID,Ship Date,Units Sold,Unit Price,Unit Cost,Total Revenue,Total Cost,Total Profit\r\nAustralia and Oceania,Solomon Islands,Baby Food,Online,C,2/4/2015,547995746,2/21/2015,2974,255.28,159.42,759202.72,474115.08,285087.64\r\n"
]

 

2 correct records

INFO  2020-06-04 16:54:59,983 [[MuleRuntime].io.07: [batch-job-sample].batch-job-batch-job-sample_Batch_Job-work-manager @72b2fd47] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: processing Region,Country,Item Type,Sales Channel,Order Priority,Order Date,Order ID,Ship Date,Units Sold,Unit Price,Unit Cost,Total Revenue,Total Cost,Total Profit
Sub-Saharan Africa,Burkina Faso,Vegetables,Online,H,7/17/2012,871543967,7/27/2012,8082,154.06,90.93,1245112.92,734896.26,510216.66

INFO  2020-06-04 16:54:59,987 [[MuleRuntime].io.07: [batch-job-sample].batch-job-batch-job-sample_Batch_Job-work-manager @72b2fd47] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: processing Region,Country,Item Type,Sales Channel,Order Priority,Order Date,Order ID,Ship Date,Units Sold,Unit Price,Unit Cost,Total Revenue,Total Cost,Total Profit
Sub-Saharan Africa,Republic of the Congo,Personal Care,Offline,M,7/14/2015,770463311,8/25/2015,6070,81.73,56.67,496101.10,343986.90,152114.20

 

Log to tell Batch_Step_Process is finished processing all records

INFO  2020-06-04 16:55:00,041 [[MuleRuntime].io.07: [batch-job-sample].batch-job-batch-job-sample_Batch_Job-work-manager @72b2fd47] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] com.mulesoft.mule.runtime.module.batch.internal.DefaultBatchStep: Step Batch_Step_Process finished processing all records for instance 06bf73b0-a656-11ea-a5d6-9aaf65ed66d8 of job batch-job-sample_Batch_Job

 

1st failed records found for which the exception was raised

INFO  2020-06-04 16:55:00,045 [[MuleRuntime].io.08: [batch-job-sample].batch-job-batch-job-sample_Batch_Job-work-manager @72b2fd47] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: failed record found Region,Country,Item Type,Sales Channel,Order Priority,Order Date,Order ID,Ship Date,Units Sold,Unit Price,Unit Cost,Total Revenue,Total Cost,Total Profit
Europe,Russia,Office Supplies,Offline,L,5/2/2014,341417157,5/8/2014,1779,651.21,524.96,1158502.59,933903.84,224598.75

 

Batch Aggregator for DB insert – we can see 2 last records

INFO  2020-06-04 16:55:00,045 [[MuleRuntime].cpuIntensive.04: [batch-job-sample].batch-job-sampleFlow.CPU_INTENSIVE @59dea51b] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: insert into DB  [
  "Region,Country,Item Type,Sales Channel,Order Priority,Order Date,Order ID,Ship Date,Units Sold,Unit Price,Unit Cost,Total Revenue,Total Cost,Total Profit\r\nSub-Saharan Africa,Burkina Faso,Vegetables,Online,H,7/17/2012,871543967,7/27/2012,8082,154.06,90.93,1245112.92,734896.26,510216.66\r\n",
  "Region,Country,Item Type,Sales Channel,Order Priority,Order Date,Order ID,Ship Date,Units Sold,Unit Price,Unit Cost,Total Revenue,Total Cost,Total Profit\r\nSub-Saharan Africa,Republic of the Congo,Personal Care,Offline,M,7/14/2015,770463311,8/25/2015,6070,81.73,56.67,496101.10,343986.90,152114.20\r\n"
]

 

2nd failed record

INFO  2020-06-04 16:55:00,047 [[MuleRuntime].io.08: [batch-job-sample].batch-job-batch-job-sample_Batch_Job-work-manager @72b2fd47] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: failed record found Region,Country,Item Type,Sales Channel,Order Priority,Order Date,Order ID,Ship Date,Units Sold,Unit Price,Unit Cost,Total Revenue,Total Cost,Total Profit
Europe,Angola,Household,Offline,M,4/23/2011,135425221,4/27/2011,4187,668.27,502.54,2798046.49,2104134.98,693911.51

 

Log to tell Batch_Step_Failed_Records is finished processing all failed records

INFO  2020-06-04 16:55:00,157 [[MuleRuntime].io.07: [batch-job-sample].batch-job-batch-job-sample_Batch_Job-work-manager @72b2fd47] [event: ] com.mulesoft.mule.runtime.module.batch.internal.DefaultBatchStep: Step Batch_Step_Failed_Records finished processing all records for instance 06bf73b0-a656-11ea-a5d6-9aaf65ed66d8 of job batch-job-sample_Batch_Job

 

Log for email which is sent for all failed records – 2 records

INFO  2020-06-04 16:55:01,174 [[MuleRuntime].cpuIntensive.06: [batch-job-sample].batch-job-sampleFlow.CPU_INTENSIVE @59dea51b] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: sending email for records [
  [
    {
      "Region": "Europe",
      "Country": "Russia",
      "Item Type": "Office Supplies",
      "Sales Channel": "Offline",
      "Order Priority": "L",
      "Order Date": "5/2/2014",
      "Order ID": "341417157",
      "Ship Date": "5/8/2014",
      "Units Sold": "1779",
      "Unit Price": "651.21",
      "Unit Cost": "524.96",
      "Total Revenue": "1158502.59",
      "Total Cost": "933903.84",
      "Total Profit": "224598.75"
    }
  ],
  [
    {
      "Region": "Europe",
      "Country": "Angola",
      "Item Type": "Household",
      "Sales Channel": "Offline",
      "Order Priority": "M",
      "Order Date": "4/23/2011",
      "Order ID": "135425221",
      "Ship Date": "4/27/2011",
      "Units Sold": "4187",
      "Unit Price": "668.27",
      "Unit Cost": "502.54",
      "Total Revenue": "2798046.49",
      "Total Cost": "2104134.98",
      "Total Profit": "693911.51"
    }
  ]
]

 

Logs for On Complete phase and completion of Batch processing

INFO  2020-06-04 16:55:02,233 [[MuleRuntime].io.08: [batch-job-sample].batch-management-work-manager @71b08f1d] [event: ] com.mulesoft.mule.runtime.module.batch.internal.engine.DefaultBatchEngine: Starting execution of onComplete phase for instance 06bf73b0-a656-11ea-a5d6-9aaf65ed66d8 of job batch-job-sample_Batch_Job
INFO  2020-06-04 16:55:02,236 [[MuleRuntime].io.08: [batch-job-sample].batch-management-work-manager @71b08f1d] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: summary report - "{\n  \"onCompletePhaseException\": null,\n  \"loadingPhaseException\": null,\n  \"totalRecords\": 9,\n  \"elapsedTimeInMillis\": 1318,\n  \"failedOnCompletePhase\": false,\n  \"failedRecords\": 2,\n  \"loadedRecords\": 9,\n  \"failedOnInputPhase\": false,\n  \"successfulRecords\": 7,\n  \"inputPhaseException\": null,\n  \"processedRecords\": 9,\n  \"failedOnLoadingPhase\": false,\n  \"batchJobInstanceId\": \"06bf73b0-a656-11ea-a5d6-9aaf65ed66d8\"\n}"
INFO  2020-06-04 16:55:02,236 [[MuleRuntime].io.08: [batch-job-sample].batch-management-work-manager @71b08f1d] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] com.mulesoft.mule.runtime.module.batch.internal.engine.DefaultBatchEngine: Finished execution of onComplete phase for instance 06bf73b0-a656-11ea-a5d6-9aaf65ed66d8 of job batch-job-sample_Batch_Job
INFO  2020-06-04 16:55:02,236 [[MuleRuntime].io.08: [batch-job-sample].batch-management-work-manager @71b08f1d] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] com.mulesoft.mule.runtime.module.batch.internal.engine.DefaultBatchEngine: Finished execution for instance '06bf73b0-a656-11ea-a5d6-9aaf65ed66d8' of job 'batch-job-sample_Batch_Job'. Total Records processed: 9. Successful records: 7. Failed Records: 2
INFO  2020-06-04 16:55:02,237 [[MuleRuntime].io.08: [batch-job-sample].batch-management-work-manager @71b08f1d] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] com.mulesoft.mule.runtime.module.batch.internal.engine.DefaultBatchEngine: 
****************************************************************************************************************************************************************
*             - - + Exception Type + - -             *                  - - + Step + - -                  *                 - - + Count + - -                  *
****************************************************************************************************************************************************************
* com.mulesoft.mule.runtime.module.batch.exception.B * Batch_Step_Process                                 * 2                                                  *
****************************************************************************************************************************************************************

INFO  2020-06-04 16:55:02,260 [[MuleRuntime].io.08: [batch-job-sample].batch-management-work-manager @71b08f1d] [event: 06bf4ca0-a656-11ea-a5d6-9aaf65ed66d8] com.mulesoft.mule.runtime.module.batch.internal.engine.DefaultBatchEngine: Finished execution for instance '06bf73b0-a656-11ea-a5d6-9aaf65ed66d8' of job 'batch-job-sample_Batch_Job'. Total Records processed: 9. Successful records: 7. Failed Records: 2

 

Sample CSV – Sample

Sample mule application – batch-job-sample

More detail on Batch Job – https://docs.mulesoft.com/mule-runtime/4.3/batch-processing-concept

  
Thank you for taking out time to read the above post. Hope you found it useful. In case of any questions, feel free to comment below. Also, if you are keen on knowing about a specific topic, happy to explore your recommendations as well.
 
For any latest updates or posts on our website, you can follow us on LinkedIn. Look forward to connecting with you there.
Share this:
Subscribe
Notify of
guest
4 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
kamuzu
kamuzu
3 years ago

Thanks for your examples! They are very helpful.

0
Reply
Saurav
Saurav
3 years ago

Very good material ever I have seen

0
Reply
sucheta patra
sucheta patra
3 years ago

Nice document. But I think I have noticed if I run the same again .its not processing. is it so?

0
Reply
Ani
Ani
3 years ago

In the first batch step aggregator size =5 , but in the diagram it says insert 10 records in logger component. Am I missing something or it needs to be corrected?

0
Reply