Essential Kafka Client Configurations for Production

batch.size= e.g. 900000: (Default: 16KiB) We believe that this value is too small for most use cases. Increase it to up to 1MB. This allows your producer to send more data in a single network request, reducing overhead and improving throughput.

linger.ms=5, 10 or 100: (Default: 0ms) This setting works hand-in-hand with batch.size. It tells the producer to wait up to a certain amount of time for a batch to fill up before sending it. Increasing this to a few milliseconds can significantly improve throughput without a major impact on latency.

compression.type=lz4 or zstd: (Default: none) Compressing your messages before they are sent to Kafka can save network bandwidth and improve throughput.

transactional.id={hostname}: This setting is only needed if you are using producer transactions. The transactional.id must be unique to each producer instance but should be persistent over restarts. When a producer instance restarts, Kafka uses this ID to prevent "zombie" producers from committing transactions. In Kubernetes, this is a key reason to use StatefulSets for your producers, as they provide a stable identity.

partitioner=murmur2_random: (Default: Murmur2Partitioner in Java, consistent_random in librdkafka) The partitioner determines which partition a message is sent to. If you are using both Java and non-Java producers, please ensure that all producers have the partitioner set to murmur2_random to ensure consistent partitioning.

group.instance.id={hostname}: Makes sure that a consumer "remembers" its previous assignment after a restart and thus, avoids expensive rebalances. But: The restart must survive the session.timeout.ms.

session.timeout.ms= e.g. 60000: (default: 30000) This setting defines the maximum time a consumer can be unavailable before it is considered failed and the consumer group is rebalanced. Increase this to a value where a consumer can easily detect its failure, shut down, and restart the pod. But only together with group.instance.id.

group.protocol=consumer: (default: classic) Use the Next Generation of the Consumer Rebalance Protocol (See KIP-848 for more information). Requires Kafka 4.0 or higher. Makes rebalancing much faster!

client.rack={dc-name}: If you are running your Kafka cluster and your clients in a multi-availability zone (AZ) cloud environment, you can set the client.rack on both the brokers and the consumers. This allows the consumer to prefer reading from a replica in the same AZ, which can significantly reduce inter-AZ data transfer costs.

num partitions=12: (default: 1) 12 is the magic number for the partition count. Read this article for more information.

replication-factor=3: (default: 1) The number of times a partition’s data is replicated across brokers. 3 is the gold standard for most production environments, providing high availability at a reasonable cost.

min.insync.replicas=2: (default: 1) How many in-sync replicas must be available for producers to write to a partition. With a replication factor of 3, setting this to 2 means, that one broker can fail without affecting production and another one can fail without loosing data.

cleanup.policy: (default: delete) This defines how Kafka handles old messages. The default is delete, which means messages are deleted after a set period. For use cases like change data capture or event sourcing, you might use compact, which removes old messages for the same key.

retention.hours: (default: 168 hours (7 days)) This setting controls how long Kafka retains messages before they are deleted. The default is 7 days, but you should adjust this based on your business requirements.