Dataweave Streaming In Mule 4

 

In this tutorial we will demonstrate how we can use dataweave streaming in mule 4 which is useful to process the large files in quickly without any memory issues which consume less resources compare to old way of reading and writing big documents.

Supported Format

  • JSON
  • XML
  • CSV

Limitations

  • When we select the deferred=true and if there is any error occurred in transform component where you set this property then error will not be thrown from that transform component
  • Streaming does not support random access

How to enable streaming

There are below configuration that we need to perform to enable streaming

  1. Streaming property: To read source data as stream
  2. Deferred property: To pass the output stream to message processor
  3. We can enable streaming with setting outputMimeType with datasource which can be a listener, File Listener, FTP listener

Now we will create a sample application where we are going to read the 1 Million records from CSV using file operation and transform the data to application/json and write the same to folder.

We will perform the operation with streaming and without streaming and see how much time it has taken to process the records and convert to json format and compare the result

 

Create a sample project and drag and drop File Listener and configure the streaming

Dataweave Streaming In Mule 4

 

<file:listener doc:name="On New or Updated File" doc:id="273e2ecc-22a5-47db-bf1f-7656402944c2"
directory="C:\data" moveToDirectory="C:\dev\src" autoDelete="true" outputMimeType="application/csv; streaming=true">
<scheduling-strategy >
<fixed-frequency frequency="15" timeUnit="SECONDS"/>
</scheduling-strategy>
</file:listener>

Perform the Transform to convert the data to JSON format and set deferred=true

Dataweave Streaming In Mule 4

 

%dw 2.0
output application/json deferred=true
---
Payload

Configure the write operation to write the json file

 

Deploy the project and write the sample file in Data folder , file will be processed automatically to destination, we can see below 1 M records csv file is processed and converted to JSON in 11 Seconds when the streaming is ON

 

Now we will deploy the project without streaming and deploy the application, we can see the processing time to convert the file is 31 Seconds

 

Sample application: DataweaveStreaming Sample

 

  
Thank you for taking out time to read the above post. Hope you found it useful. In case of any questions, feel free to comment below. Also, if you are keen on knowing about a specific topic, happy to explore your recommendations as well.
 
For any latest updates or posts on our website, you can follow us on LinkedIn. Look forward to connecting with you there.


Share this:
Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments