Message Broker: Kafka
See the Message Broker page for a description of how message queues work by default in Smile CDR.
Rather than using a JMS message broker, you can configure Smile CDR to use Kafka instead.
Set module.clustermgr.config.messagebroker.type to KAFKA in the Cluster Manager to have Smile CDR use Kafka instead of a JMS message broker for subscription processing.
Smile CDR uses the following configuration properties for Kafka configuration:
module.clustermgr.config.messagebroker.type=KAFKA
module.clustermgr.config.kafka.bootstrap_address=localhost:9092
module.clustermgr.config.kafka.group_id=smilecdr
module.clustermgr.config.kafka.auto_commit=false
module.clustermgr.config.kafka.validate_topics_exist_before_use=false
module.clustermgr.config.kafka.ack_mode=MANUAL
The values above for the module.clustermgr.config.kafka.* properties are the defaults, so you don't need to include these properties in your config if you are using the default value.
bootstrap_address is a comma-separated list of host and port pairs that are the addresses of the Kafka brokers in a "bootstrap" Kafka cluster that a Kafka client connects to initially to bootstrap itself.
group_id is the name of the kafka consumer group id used by all Smile CDR Kafka subscription consumers. It's fine to leave this as the default.
auto_commit If this property is set to true, Kafka auto-commits the offsets according to its configuration. If it is false, the Kafka Consumer Ack Mode is used to determine commit behavior. See Committing Offsets
validate_topics_exist_before_use. Set this to true if your Kafka broker is configured to prevent new topics from being automatically created (e.g. if 'auto.create.topics.enable' is set to false on the broker). When this property is set to true, Smile will prevent subscriptions from being created or updated if the delivery topic they depend on doesn't exist yet.
ack_mode specifies the Kafka consumer listener container offset commit behavior. This value is only used if Kafka Auto Commit is false. By default, cdr kafka consumers manually acknowledge every successful message processing back to the Kafka broker. This ensures that no message is ever processed twice. However it adds processing overhead. If downstream consumers can process the same message twice, then it is recommended to set kafka.auto_commit to true (in which case the kafka.ack_mode property will be ignored). See Committing Offsets for a list of options.
Like the JMS Message Broker, Kafka uses the Cluster Manager module consumers_per_matching_queue and consumers_per_delivery_queue properties to determine the number of consumers for each topic. Note that for kafka, the number of consumers should be less than or equal to the number of partitions per topic set on the broker num.partitions configuration property. See Spring Kafka Multi-Threaded Message Consumption.
For Kafka brokers, the Smile CDR channels are synonymous with Kafka topics. See Channel Names for details on how these topics will be named. Note that due to Kafka naming conventions, any - (dash) characters in channel names will be replaced with a . (dot). So for example, the Subscription matching topic in a default configuration will be named subscription.matching.Master.persistence.