Migration Pattern

Migration pattern talks about migrating data from one system to another system, mainly from older system to new system. In organization there always a need of migrating data from older system to new system so that system can align with the new versions. Migration pattern is also know as unidirectional migration pattern as it only talk about transferring data from source system to target system and not vice-versa. Below we can see one of the use case where organization wants to move their older version to new version.

We can define migration in following steps

• Source system/application where data reside and need to be migrated to target system.
• Full load or Delta load – Like to move all source data to target or partial data.
• Criteria or condition which define the eligible data in source system e.g. customer modification date is greater than yesterday.
• Enrichment and transformation as per target system.
  • Enrichment – pulling more information which is not available in source system
  • Transformation – transforming it to desired target format which is acceptable by the target system e.g. Full name instead of First name and Last name
• End result
  • A report to show the status of current migration

Migration in MuleSoft

Batch Jobs is the widely used activity in MuleSoft for migrating data from one system to another as it provide highest throughput because of parallel execution.

How Batch Jobs can be constructed

Source to hold

• • Scheduling logic along with calling Source System API to retrieve data
  • Polling logic – if source data/event is coming directly from the source system

Enrichment and Transformation – Batch Step – Accept Policy (ALL)

• • Enrichment – for a record we call other APIs to get additional details
  • Transformation – To convert the source system data required by the target system

Error Handling – Batch Step – Accept Policy (ONLY_FAILURES)

• • To do something at record level failure
  • To do certain activity for all the failed record only once – then add aggregator activity

Reporting – On Complete

• • Send the migration status details as report

Below is how the batch job can be constructed

With Scheduler

With Polling

MuleSoft API led approach in Migration

As the migration is between system and there is not point of exposing any interface to outer world. We can have below API Led approach for migration

One Process Layer application which will hold following

• • Scheduling /polling logic
  • Call to Source System API
  • Do the required transformation and enrichment
  • Call the Target System API
  • Create the report if required
  • Handle any error occur while data migration

Two System Layer applications

• • Source System API – call the source system to get relevant data which need to be migrated
  • Target System API – call the target system to insert/update the data

Sample Application: migration-pattern-sample

Reference blog – data-integration-patterns-migration