Kafka Endless Rebalancing When Adding New Instance

[SzymonZasada]

I'm experiencing an endless rebalancing loop when adding new instances. The consumer group never stabilizes and keeps rebalancing continuously.
I can only use one instance, regardless of whether I have 1-10 concurrency per instance. Each additional instance (above 1) results in infinite rebalancing.

I pool 200 messages at a time. It takes me about 50-60 seconds max to process them all.
-20 topics each 30 partitions

Environment:
Spring Boot 3.5.8 with Spring Kafka
30 partitions per topic
concurrency=10 per instance
Running in Docker with graceful shutdown working correctly

Errors:
Request joining group due to: group is already rebalancing

Kafka config:

`@EnableKafka
@configuration
@slf4j
public class KafkaConfig {

private static final int POLL_TIMEOUT_MS = 150_000;  // 2.5 min

@Value("${kafka.bootstrap-servers}")
private String bootstrapServers;

//producer
@Bean
public KafkaTemplate<String, String> kafkaTemplate() {
    return new KafkaTemplate<>(producerFactory());
}

@Bean
public ProducerFactory<String, String> producerFactory() {
    Map<String, Object> configProps = new HashMap<>();
    configProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
    configProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
    configProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);

    configProps.put(ProducerConfig.RETRIES_CONFIG, new DefaultKafkaConfig().getMaxRetries());
    configProps.put(ProducerConfig.RETRY_BACKOFF_MS_CONFIG, 1000);
    configProps.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
    configProps.put(ProducerConfig.ACKS_CONFIG, "all");

    configProps.put(ProducerConfig.INTERCEPTOR_CLASSES_CONFIG,
            LoggingProducerInterceptor.class.getName());

    return new DefaultKafkaProducerFactory<>(configProps);
}

//consumer
@Bean
public ConsumerFactory<String, String> consumerFactory() {
    Map<String, Object> configProps = new HashMap<>();
    configProps.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
    configProps.put(ErrorHandlingDeserializer.KEY_DESERIALIZER_CLASS, ErrorHandlingDeserializer.class);
    configProps.put(ErrorHandlingDeserializer.VALUE_DESERIALIZER_CLASS, ErrorHandlingDeserializer.class);
    configProps.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
    configProps.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);

    configProps.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 200);

    configProps.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG, "org.apache.kafka.clients.consumer.CooperativeStickyAssignor");


    configProps.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, 10_000);
    configProps.put(ConsumerConfig.HEARTBEAT_INTERVAL_MS_CONFIG, 3_000);
    configProps.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, POLL_TIMEOUT_MS);

    configProps.put(ConsumerConfig.DEFAULT_API_TIMEOUT_MS_CONFIG, 300_000);
    configProps.put(ConsumerConfig.REQUEST_TIMEOUT_MS_CONFIG, 90_000);

    return new DefaultKafkaConsumerFactory<>(configProps);
}

@Bean
public ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactory(KafkaMdcInterceptor kafkaMdcInterceptor) {
    ConcurrentKafkaListenerContainerFactory<String, String> factory =
            new ConcurrentKafkaListenerContainerFactory<>();
    factory.setConsumerFactory(consumerFactory());

    int maxRetries = new DefaultKafkaConfig().getMaxConsumerRetries();
    factory.setCommonErrorHandler(new LoggingErrorHandler(new FixedBackOff(500L, maxRetries - 1)));

    configureFactory(factory, kafkaMdcInterceptor);

    return factory;
}

@Bean
public ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactoryNoRetry(KafkaMdcInterceptor kafkaMdcInterceptor) {
    ConcurrentKafkaListenerContainerFactory<String, String> factory =
            new ConcurrentKafkaListenerContainerFactory<>();
    factory.setConsumerFactory(consumerFactory());

    // Without retry - improtant
    factory.setCommonErrorHandler(new LoggingErrorHandler(new FixedBackOff(0L, 0L)));

    configureFactory(factory, kafkaMdcInterceptor);

    return factory;
}


private void configureFactory(ConcurrentKafkaListenerContainerFactory<String, String> factory,
                              KafkaMdcInterceptor kafkaMdcInterceptor) {
    SimpleAsyncTaskExecutor executor = new SimpleAsyncTaskExecutor();
    executor.setVirtualThreads(true);

    factory.getContainerProperties().setShutdownTimeout((long) POLL_TIMEOUT_MS);
    factory.getContainerProperties().setStopImmediate(false);


    factory.getContainerProperties().setListenerTaskExecutor(executor);
    factory.getContainerProperties().setDeliveryAttemptHeader(true);
    factory.setRecordInterceptor(kafkaMdcInterceptor);
}

}`

[artembilan]

Please, describe what is "Adding New Instance" in your meaning?
Any chances that you can try your solution with Spring Boot 4.0 where we use Apache Kafka 4.0 already?
I mean it is not clear what concern you claim exactly about Spring at all since all the rebalance functionality is handled by the Kafka itself.
Thanks

[SzymonZasada]

Hi,
Thanks for your questions. Let me clarify:

Adding New Instance means:
Creating another instance of my consumer application - essentially scaling up my deployment. For example, if I currently have one Spring Boot application running with consumer group "my-consumer-group", and I deploy a second instance of the same application (same consumer group), that's "adding a new instance."

Currently, I can only run ONE instance successfully. When I try to scale to TWO instances (or more), the consumer group enters an endless rebalancing loop and never stabilizes. So I'm stuck with a single instance, even though I have 600 partitions (20 topics × 30 partitions each) that could theoretically be distributed across multiple instances.

Could you take a look at my Kafka configuration? Maybe there's something misconfigured that's causing this issue. I've shared my config in the original post, but I'd appreciate if you could spot any potential problems.

Regarding Spring Boot 4.0.0: I haven't upgraded yet, but I'm planning to move to Spring Boot 4.0 within about a week or so. I can test the solution there once I upgrade.

[artembilan]

OK. Since you have all the beans manually, that means you don't use Spring Boot auto-configuration.
So, perhaps we can narrow that down just to Spring Kafka.
I don't see in your code where you specify the mentioned my-consumer-group, but that probably does not matter since according to the rebalance outcome it is in the action.
I see that you use CooperativeStickyAssignor.
Any chances you can try with other variants?

And the most reliable proof would be if you could replicate such a rebalance problem even without Spring at all, just with plain Apache Kafka client.
I say this to claim again that Spring does not do anything rebalance-related, everything is handled by the Kafka functionality itself.
When we narrow the problem down, we may try to search Apache Kafka issue tracker for similar reports.

Thanks

[SzymonZasada]

Hi @artembilan

Let me clarify what I mean by adding instances in the Kafka consumer context:

How Kafka Consumer Groups Work:

When you have a Kafka consumer group (e.g., my-consumer-group), all consumers in that group share the work of reading from the topic's partitions. If I have 30 partitions and 1 consumer instance, that single instance handles all 30 partitions. When I add a second instance to the same consumer group, Kafka should automatically rebalance - for example, each instance would then handle 15 partitions. (OFC I have concurency=10 for consumer)

My Goal:
I want to scale horizontally by adding more application instances (same code, same consumer group, same topics) to increase throughput. This is a standard Kafka scaling pattern.

The Problem:
When I start instance #2, Kafka triggers a rebalance (expected), but then it never completes - it keeps rebalancing endlessly. The error group is already rebalancing repeats continuously, and neither instance can consume messages.

Do you see any configuration issues that could cause this?
Could the combination of 20 topics × 30 partitions × concurrency=10 be creating too many consumers in the group?

[artembilan]

Thank you for detailed explanation about consumer group management in Apache Kafka!

And I'm stating again: Spring does nothing with rebalance.
If there is a problem with rebalance, it is better to go to the proper place to raise that concern.
And as far as I know this project, we just cannot help you solve your problem from Spring perspective, because the whole group management is done on Apache Kafka side, internally in their client or further in the broker.

I gave you already some thoughts what to try from the Kafka configuration perspective.
Another idea is to look into this property yet: https://docs.confluent.io/platform/current/installation/configuration/broker-configs.html#group-initial-rebalance-delay-ms.

You also may start looking into Apache Kafka 4.x migration where they have a new rebalance algorithm.

[sobychacko]

Hi @SzymonZasada, have you had a chance to upgrade to Spring Boot 4.0.x / Spring Kafka 4.0.x (Apache Kafka 4.0.x/4.1.x) yet? Kafka 4.0.0 introduced the new server-side consumer group protocol which replaces the client-side rebalance strategies like CooperativeStickyAssignor. With 600 partitions spread across your consumers, the old client-side rebalance protocol can struggle to converge -- which aligns with the symptoms you're seeing. The new protocol handles assignment server-side and should resolve this class of issue. We'd recommend trying that before investigating further.