Migration Pattern

 

Migration pattern talks about migrating data from one system to another system. In organization there always a need of migrating data from one system to other system so that system can work in sync. In below if any Sales happen in Customer Web Portal then CRM application should also know about the sales. Migration pattern is also know as unidirectional migration pattern as it only talk about transferring data from source system to target system and not vice-versa.

 

Migration Pattern

 

We can define migration in following steps

  • Source system/application where data reside and need to be migrated to target system.
  • Frequency at which you want to migrate out data e.g. Real time – every second or once in day or any custom frequency.
  • Criteria or condition which define the eligible data in source system e.g. customer modification date is greater than yesterday.
  • Enrichment and transformation as per target system.
    • Enrichment – pulling more information which is not available in source system
    • Transformation – transforming it to desired target format which is acceptable by the target system e.g. Full name instead of First name and Last name
  • End result
    • A report to show the status of current migration

 

Migration in MuleSoft

Batch Jobs is the widely used activity in MuleSoft for migrating data from one system to another as it provide highest throughput because of parallel execution.

 

How Batch Jobs can be constructed

Source to hold

    • Scheduling logic along with calling Source System API to retrieve data
    • Polling logic – if source data/event is coming directly from the source system

Enrichment and Transformation – Batch Step – Accept Policy (ALL)

    • Enrichment – for a record we call other APIs to get additional details
    • Transformation – To convert the source system data required by the target system

Error Handling – Batch Step – Accept Policy (ONLY_FAILURES)

    • To do something at record level failure
    • To do certain activity for all the failed record only once – then add aggregator activity

Reporting – On Complete

    • Send the migration status details as report

Below is how the batch job can be constructed

With Scheduler

Migration Pattern

 

With Polling

Migration Pattern

 

MuleSoft API led approach in Migration

As the migration is between system and there is not point of exposing any interface to outer world. We can have below API Led approach for migration

 

One Process Layer application which will hold following

    • Scheduling /polling logic
    • Call to Source System API
    • Do the required transformation and enrichment
    • Call the Target System API
    • Create the report if required
    • Handle any error occur while data migration

Two System Layer applications

    • Source System API – call the source system to get relevant data which need to be migrated
    • Target System API – call the target system to insert/update the data

 

Sample Application: migration-pattern-sample

Reference blogdata-integration-patterns-migration

  
Thank you for taking out time to read the above post. Hope you found it useful. In case of any questions, feel free to comment below. Also, if you are keen on knowing about a specific topic, happy to explore your recommendations as well.
 
For any latest updates or posts on our website, you can follow us on LinkedIn. Look forward to connecting with you there.


Share this:
Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments