Consume Latest Message From Kafka Topic

In this Tutorials, We have created integration between Kafka topic and Mulesoft. We will publish the message on the topic and consume the latest message published on to the topic. We will then parse the message and write to a file.

If you want to read more about kafka, please visit kafka connector for more information.

Prerequisite:

• Should have Kafka Topic already set up (Topic Name and Broker URL)
• Kafka Module should be added in palette( Please visit this url to add kafka module in Anypoint studio)

Ref: https://docs.mulesoft.com/kafka-connector/4.1/kafka-connector-reference

Drag and drop kafka publish operation from mule palette

Configure the topic name(visit this url to create the kafka topic) and payload

Configure the producer connection details and the bootstrap server Url

Currently I have kept all the default values as is and Producer Acknowledge Mode=ALL

Configure the Kafka Message Listener to consume the message from the topic as shown in the picture below

Configure the consumer connection details as shown below

If you want to consume only the latest message from kafka topic, Please set “Auto Offset Reset”  to  “LATEST” and keep other values as default.

If you want to consume all the message published from kafka topic Please set “Auto Offset Reset”  to  “EARLIEST” and keep other values as default.

Note: Kafka stores the messages for 7 Days

Convert binary Data to JSON and write the payload to file using write file connector.

Transform the message for further processing

%dw 2.0
output application/json
---
payload map{
  CustomerName:$.Firstname,
  CustomerID:$.Customerid
}

Deploy the application and test to consume the latest message only.

Transformed payload written successfully in file

we have attached the SOAP UI project and sample application below.

Sample application: kafka consumer sample application

Soapui project: consume message soapui project

We hope you liked the tutorials. Please let us know if you need any other information on this. Happy reading and Learning!