Migration pattern talks about migrating data from one system to another system. In organization there always a need of migrating data from one system to other system so that system can work in sync. In below if any Sales happen in Customer Web Portal then CRM application should also know about the sales. Migration pattern is also know as unidirectional migration pattern as it only talk about transferring data from source system to target system and not vice-versa.
We can define migration in following steps
- Source system/application where data reside and need to be migrated to target system.
- Frequency at which you want to migrate out data e.g. Real time – every second or once in day or any custom frequency.
- Criteria or condition which define the eligible data in source system e.g. customer modification date is greater than yesterday.
- Enrichment and transformation as per target system.
- Enrichment – pulling more information which is not available in source system
- Transformation – transforming it to desired target format which is acceptable by the target system e.g. Full name instead of First name and Last name
- End result
- A report to show the status of current migration
Migration in MuleSoft
Batch Jobs is the widely used activity in MuleSoft for migrating data from one system to another as it provide highest throughput because of parallel execution.
How Batch Jobs can be constructed
Source to hold
- Scheduling logic along with calling Source System API to retrieve data
- Polling logic – if source data/event is coming directly from the source system
Enrichment and Transformation – Batch Step – Accept Policy (ALL)
- Enrichment – for a record we call other APIs to get additional details
- Transformation – To convert the source system data required by the target system
Error Handling – Batch Step – Accept Policy (ONLY_FAILURES)
- To do something at record level failure
- To do certain activity for all the failed record only once – then add aggregator activity
Reporting – On Complete
- Send the migration status details as report
Below is how the batch job can be constructed
MuleSoft API led approach in Migration
As the migration is between system and there is not point of exposing any interface to outer world. We can have below API Led approach for migration
One Process Layer application which will hold following
- Scheduling /polling logic
- Call to Source System API
- Do the required transformation and enrichment
- Call the Target System API
- Create the report if required
- Handle any error occur while data migration
Two System Layer applications
- Source System API – call the source system to get relevant data which need to be migrated
- Target System API – call the target system to insert/update the data
Sample Application: migration-pattern-sample
Reference blog – data-integration-patterns-migration