Skip to content

Allow to distribute consumer group messages evenly when running multiple servers with multiple consumers each (set consumer priority to thread idx ?) #3122

@jonenst

Description

@jonenst

Describe the issue
When using multiple consumers on multiple servers, messages in a consumer groups are not evenly distributed across servers.

To Reproduce
Steps to reproduce the behavior:
using a consumer group where message handling is a computationaly intensive operation (maybe 10s of cpu), start 2 servers with 2 consumers each (e.g. SMLC concurrency=2). Publish 2 messages. Notice that both message are handled by the same server going 200% cpu, whereas the other server is idle. Also, even for small loads, servers must have enough ram to handle all the messages in parallel.

Version of the framework
2023.0.1

Expected behavior
Allow to setup evenly distributed messages across servers each running multiple consumers
For example, if each thread could set it's consumerpriority to it's index, rabbitmq would only distribute messages to low threads when higher threads are actually running

Or maybe another idea, like dynamically creating consumers under load ? But it's harder to control when its scaling down.

or create manually N bindings each with a concurrency of 1 and running the same consumer bean ?

Additional context
related issue in spring-amqp spring-projects/spring-amqp#3092

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions