Message Broker: Kafka

See the Message Broker page for a description of how message queues work by default in Smile CDR.

The Apache Kafka broker is the recommended broker option, particularly for large scale deployments of Smile CDR.

Set module.clustermgr.config.messagebroker.type to KAFKA in the Cluster Manager to have Smile CDR use Kafka instead of a JMS message broker for subscription processing.

Smile CDR uses the following configuration properties for Kafka configuration:

module.clustermgr.config.messagebroker.type=KAFKA
module.clustermgr.config.kafka.bootstrap_address=localhost:9092
module.clustermgr.config.kafka.group_id=smilecdr
module.clustermgr.config.kafka.auto_commit=false
module.clustermgr.config.kafka.validate_topics_exist_before_use=false
module.clustermgr.config.kafka.ack_mode=MANUAL
module.clustermgr.config.kafka.producer.auto_flush=true

The values above for the module.clustermgr.config.kafka.* properties are the defaults, so you don't need to include these properties in your config if you are using the default value.

bootstrap_address is a comma-separated list of host and port pairs that are the addresses of the Kafka brokers in a "bootstrap" Kafka cluster that a Kafka client connects to initially to bootstrap itself.

group_id is the name of the kafka consumer group id used by all Smile CDR Kafka subscription consumers. It's fine to leave this as the default.

auto_commit If this property is set to true, Kafka auto-commits the offsets according to its configuration. If it is false, the Kafka Consumer Ack Mode is used to determine commit behavior. See Committing Offsets

validate_topics_exist_before_use. Set this to true if your Kafka broker is configured to prevent new topics from being automatically created (e.g. if 'auto.create.topics.enable' is set to false on the broker). When this property is set to true, Smile will prevent subscriptions from being created or updated if the delivery topic they depend on doesn't exist yet.

auto_flush controls whether the Kafka producer flushes after every send. The default true (introduced in Release 2025.05) narrows the message-loss window before broker acknowledgement; in practice it prevents producer-side batching for sequential sends and neutralizes Kafka's linger.ms and batch.size properties. When set to false, these Kafka properties will be used and records may be batched. In exchange, records may sit longer in the producer's in-memory accumulator, increasing the window for loss if the JVM crashes before acknowledgement.

ack_mode specifies the Kafka consumer listener container offset commit behavior. This value is only used if Kafka Auto Commit is false. By default, cdr kafka consumers manually acknowledge every successful message processing back to the Kafka broker. This ensures that no message is ever processed twice. However it adds processing overhead. If downstream consumers can process the same message twice, then it is recommended to set kafka.auto_commit to true (in which case the kafka.ack_mode property will be ignored). See Committing Offsets for a list of options.

Like the JMS Message Broker, Kafka uses the Cluster Manager module consumers_per_matching_queue and consumers_per_delivery_queue properties to determine the number of consumers for each topic. Note that for kafka, the number of consumers should be less than or equal to the number of partitions per topic set on the broker num.partitions configuration property. See Spring Kafka Multi-Threaded Message Consumption.

Kafka Topic Names

For Kafka brokers, the Smile CDR channels are synonymous with Kafka topics. See Channel Names for details on how these topics will be named.

Note that prior to Release 2023.02.R01 all - (hyphen) characters were replaced with a . (period) in Smile CDR generated channel names. So for example, the Subscription matching topic in a default configuration was named subscription.matching.Master.persistence.

In Release 2023.02.R01 and onwards, this functionality can be disabled using the Kafka Replace Hyphens with Periods property.

Message Ordering with Partition Keys

Kafka guarantees message ordering within a partition. To ensure related messages are processed in order, assign the same partition key to messages that must be processed sequentially.

When a message is published with a partition key, Kafka hashes the key to determine which partition receives the message. All messages with the same key hash to the same partition, ensuring they are consumed in order.

Partition Key Selection

Choose a partition key based on the ordering requirements of your data:

• Patient-centric data: Use the patient identifier (e.g., MRN, enterprise ID) as the partition key
• Non-patient-centric data: Use the resource ID or another stable identifier. For low-volume non-patient data, a shared constant key routes all such messages to a single partition.

Trade-offs: Ordering vs. Parallelism

Partition keys affect parallelism:

• More unique keys = messages distributed across more partitions = higher parallelism
• Fewer unique keys = messages concentrated in fewer partitions = stronger ordering guarantees but less parallelism

For example, using patient ID as the partition key allows parallel processing across different patients while ensuring all messages for the same patient are processed in order.

Consumer Configuration

The number of consumers should be less than or equal to the number of partitions. See the consumers_per_matching_queue and consumers_per_delivery_queue properties in Message Broker configuration.

For guidance on setting partition keys in Camel routes, see Message Keys for Ordering.

Kafka - Overriding Default Configuration Settings

If you need to override any Kafka default configuration settings, for either the Producer or Consumer, you can configure your own properties files using the Web Administration Console in the Cluster Manager "Kafka" Section. The following sections are available for override:

• Producer Config Properties
• Consumer Config Properties

If the above properties are set, the indicated files must exist in the location defined. You can also store those files in the classpath, by prefixing the filename with classpath: and then the relative location, e.g. classpath:/cdr-kafka-producer-config.properties.

NOTE: This example is using an actual file on the filesystem to override default configuration values, but you can also use a text value in the base Smile CDR configuration file, if you do not wish to use this approach.

One example of where this approach can be very useful, is when you are loading data into Smile CDR and it is causing a lot of Kafka messages to be produced, and that causes Producer or Consumer heartbeat timeouts to occur which then causes Kafka to rebalance. When Kafka rebalances, all message processing will stop for a few minutes, and it also could cause exceptions and errors to be logged in the Smile CDR Server log in a cascading failure. You should always try to avoid a Kafka rebalancing scenario. To help avoid this scenario, you can override some default configuration values in your custom properties file. We have found the following settings generally good for avoiding rebalancing issues, but they may need to be modified for specific use cases:

# 2 minutes
heartbeat.interval.ms=120000

# 5 minutes
session.timeout.ms=300000

# 60 minutes
max.poll.interval.ms=3600000

# Only grab 200 records at a time so we can process them before a timeout occurs
max.poll.records=200

Only configuration settings that you wish to override need to be included. The rest of the settings will use the default values.

Configuring Kafka with SASL

The following configuration shows an example of how to configure Smile to work with a SASL_PLAINTEXT configured Kafka broker.

	kafka.sasl.mechanism=PLAIN
	kafka.security.protocol=SASL_PLAINTEXT
	kafka.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
		username=admin_username \
		password=admin_password;