Skip Ribbon Commands
Skip to main content
Navigate Up
Sign In

Quick Launch

Average Rating:

facebook Twitter
Email
Print Bookmark Alert me when this article is updated

Feedback

How to configure PowerExchange CDC publisher to send/distribute messages to multiple partitions in a Kafka Topic?
Answer
When the Publisher is configured for “guaranteed delivery” (Connector.kafkaProducerGuaranteeDelivery =True), all messages are sent to exactly one Kafka topic and partition

To allow CDC Publisher to support multiple partitions in the targets topics, configure the following properties in the Connector configuration file, cdcPublisherKafka.cfg:
Connector.kafkaProducerGuaranteeDelivery=false
Connector.kafkaProducerPartitionID=-1
Connector.sendMaintainOrder=false

If you disable 'Guaranteed delivery' and use default -1 for kafkaProducerPartitionID, KAFKA assigns the partition (Publisher is effectively providing no partition).

When Kafka assigns the partition, it uses the message key.   If there is No message key, Kafka assigns messages in round-robin to the available partitions of a Topic.

If there is a message key, Kafka assigns partition based on the Hash key value from the messages. 

Note: If you allow CDC Publisher to write messages to multiple partitions in any target topic, the order in which changes were retrieved from the source is not maintained when writing messages to the target topics. 



More Information

Applies To
Product: PowerExchange
Problem Type: Configuration; Product Feature
User Type: Administrator; Developer
Project Phase: Configure; Implement; Optimize
Product Version:
Database: Kafka
Operating System:
Other Software:

Reference

Attachments

Last Modified Date:9/20/2019 9:19 AMID:533920
People who viewed this also viewed

Feedback

Did this KB document help you?



What can we do to improve this information (2000 or fewer characters)