Using blocking publisher confirms with concurrent publishers

[MichaelLogutov]

Hello.
We're using rabbitmq library version 5 in web applications to publish messages from request. As you may guess it's highly concurrent environment and it's quite common to have 100-500 requests per second sending a message to rabbit.

Trying to update to version 6 of the library (6.2.1 at this time) shows a lot of timeout errors while sending the message. The problem can be reproduced locally and I've attached a simple solution with two projects:

• RabbitMQ (which is using 5.2.0)
• RabbitMQ2 (which is using 6.2.1)

Also I've added simple docker compose file to spin single local rabbitmq node on 127.0.0.11

Each project contains identical code which performs this steps:

1. Opens single (static) connection
2. Creates exchange "tests" (durable with type = topic)
3. Starts 100 parallel tasks trying to publish a message to "test" exchange with publish confirm

Step 3 implemented in 3 different ways and can be switched by passing specific argument when running project:

• (default) always creates new channel for each publish and disposes it
• (per_thread) creates channel per thread (using ThreadStatic)
• (per_thread_with_lock) creates channel per thread (using ThreadStatic) but performs locking so only 1 thread opens up a channel at a time
• (single) only one channel created and every task is under single lock to accessing it

Running all tests on v5 gets me a very good performance - 7-9 msec per message (100 messages total in parallel).
Running all tests on v6 never completed without timeout errors.

Can someone please look into this? Maybe I'm doing something wrong? How to use rabbitmq v6 in high concurrent environment like web?

RabbitMQ.zip

[michaelklishin]

Thank you for your time.

Team RabbitMQ uses GitHub issues for specific actionable items engineers can work on. GitHub issues are not used for questions, investigations, root cause analysis, discussions of potential issues, etc (as defined by this team).

We get at least a dozen of questions through various venues every single day, often light on details.
At that rate GitHub issues can very quickly turn into a something impossible to navigate and make sense of even for our team. Because GitHub is a tool our team uses heavily nearly every day, the signal/noise ratio of issues is something we care about a lot.

Please post this to rabbitmq-users.

Thank you.

[michaelklishin]

Sounds like a different case of what was discussed in #876, #945.

[michaelklishin]

Some of these approaches cannot possibly work because concurrent publishing on a shared channel is explicitly not supported.

Opening a channel and enabling confirms mode on it in a critical section, but not for actual publishing, cannot possibly work either and will result in connection-level errors due to incorrect framing in RabbitMQ logs.

WaitForConfirmsOrDie is very inefficient and in this case, hardly even convenient. Streaming confirms with a handler make significantly more sense for medium or high-volume concurrent publishing.

In 6.x, library-level locking around network writes has been eliminated (or greatly reduced, in case of TLS sockets). This means such concurrent operation issues are much more visible.

SendMessageCreateChannelAlways seems safe but that's not how channels are supposed to be used.

This basic integration test is a good starting point. It does not use confirms or streaming confirms but it does demonstrate what kind of concurrent access scenarios are expected (or not).

[MichaelLogutov]

There is not a single concurrent publishing on a channel on any of the methods. Please, read the code - I reuse channels just like it's been stated in documentation. Even on "per thread" there is no path where currently executing code could use another thread's channel.

6.x does not handle even work on SendMessageCreateChannelAlways - it's simply times out almost all the time. Have you even tried to run my sample?

Your suggested integration test assuming that for using v6 application should use SetMinThreads? You can't be serious about that. You can't demand that modern microservice application to limit itself with multithreading just to use rabbitmq v6. No other library we're using in our microservices platform implies such limitations - rabbitmq 5, mssql, pgsql, cassandra, redis, consul.

There is something wrong with you library and multiple people trying to prove it to you and you guys just keep denying it and closing tickets.

[lukebakken]

@MichaelLogutov please be patient. If it seems like we don't understand your code, you should take the time to explain.
It was not immediately obvious to me that you weren't sharing channels until I saw this line -

https://github.com/lukebakken/rabbitmq-dotnet-client-959/blob/master/RabbitMQ2/Program.cs#L119-L120

Version 6.0 of this library uses the .NET Thread Pool. You must ensure that enough threads are available in the thread pool when your application starts by doing something like this:

https://github.com/lukebakken/rabbitmq-dotnet-client-959/blob/master/RabbitMQ2/Program.cs#L15-L16

With the above change, your code works fine using version 6.2.1 of this library. Similar changes fixed #860

Your suggested integration test assuming that for using v6 application should use SetMinThreads?

Please educate yourself as to what ThreadPool.SetMinThreads does: link.

@michaelklishin could you suggest a better title for this issue? Something like "Use of ThreadPool in 6.X should be more clearly documented".

@stebet now that the thread pool is being used how can we better guide users so that they do not make the mistake here and in #860? Can we detect if default ThreadPool settings are in place and throw an exception?

@MichaelLogutov here is the output of a run on my Arch Linux workstation. I have no idea why the RabbitMQ2 "avg time" totals are larger than those from RabbitMQ. I don't have time to diagnose further right now (cc @stebet ... ???)

$ ./run.sh 
Microsoft (R) Build Engine version 16.4.0+e901037fe for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  Restore completed in 23.92 ms for /home/lbakken/issues/rabbitmq/rabbitmq-dotnet-client/gh-959/repro/RabbitMQ2/RabbitMQ2.csproj.
  Restore completed in 23.92 ms for /home/lbakken/issues/rabbitmq/rabbitmq-dotnet-client/gh-959/repro/RabbitMQ/RabbitMQ.csproj.
  RabbitMQ2 -> /home/lbakken/issues/rabbitmq/rabbitmq-dotnet-client/gh-959/repro/RabbitMQ2/bin/Debug/netcoreapp3.1/RabbitMQ2.dll
  RabbitMQ -> /home/lbakken/issues/rabbitmq/rabbitmq-dotnet-client/gh-959/repro/RabbitMQ/bin/Debug/netcoreapp3.1/RabbitMQ.dll

Build succeeded.                                                                                                                                                                                                                                                                
    0 Warning(s)                                                                                                                                                                                                                                                                
    0 Error(s)

Time Elapsed 00:00:00.66
~/issues/rabbitmq/rabbitmq-dotnet-client/gh-959/repro/RabbitMQ ~/issues/rabbitmq/rabbitmq-dotnet-client/gh-959/repro
Act: SendMessageCreateChannelAlways
Sent 100 messages, avg time 2 ms, 0 errors (total_millis: 254)
Total time is 0.0459944 seconds

Act: SendMessageCreateChannelPerThread
Sent 100 messages, avg time 1 ms, 0 errors (total_millis: 101)
Total time is 0.0262683 seconds

Act: SendMessageCreateChannelPerThreadWithLock
Sent 100 messages, avg time 0 ms, 0 errors (total_millis: 71)
Total time is 0.0207761 seconds

Act: SendMessageSingleChannel
Sent 100 messages, avg time 2 ms, 0 errors (total_millis: 211)
Total time is 0.0385052 seconds

~/issues/rabbitmq/rabbitmq-dotnet-client/gh-959/repro
~/issues/rabbitmq/rabbitmq-dotnet-client/gh-959/repro/RabbitMQ2 ~/issues/rabbitmq/rabbitmq-dotnet-client/gh-959/repro
Act: SendMessageCreateChannelAlways
Sent 100 messages, avg time 56 ms, 0 errors (total_millis: 5683)
Total time is 0.0782001 seconds

Act: SendMessageCreateChannelPerThread
Sent 100 messages, avg time 24 ms, 0 errors (total_millis: 2432)
Total time is 0.0430014 seconds

Act: SendMessageCreateChannelPerThreadWithLock
Sent 100 messages, avg time 33 ms, 0 errors (total_millis: 3345)
Total time is 0.0699946 seconds

Act: SendMessageSingleChannel
Sent 100 messages, avg time 16 ms, 0 errors (total_millis: 1606)
Total time is 0.0295687 seconds

[stebet]

If highly concurrent publishing is desired, I'd suggest using channels (from System.Threading.Channels) as intermediate buffers, and create worker tasks that consume from those channels and use batch publishing if enough messages are waiting. That way you'll also reduce IO overhead by utilizing fewer connections/models more efficiently.

I'll write up a sample that could be added to the documentation.

@michaelklishin @lukebakken would it make sense to remove the "WaitForConfirms*" methods from the 7.0 version and rather document how users could utilize streaming confirms? It'd simplify a lot of logic in the current client and probably prevent incorrect usage of confirms.

[MichaelLogutov]

@lukebakken Thank you for taking time explaning and reading through my code.
About increasing SetMinThreads - maybe I didn't word myself right, but there is a big warning on the same document page about increasing min threads and how it could affect high concurrent applications - that what I was talking about. I think it's a wrong design for a library to demand such implications from app.

@stebet Would be great to see working prototype with the same throughput as v5.

You guys doing a great job and I'm sure you did a lot of work with v6, it's just I responsible for all "platform" libraries in our company and it seems even though I'm a long time rabbitmq library user I can't come up with any solution that can utilize v6 to be as effective as it v5 was in high concurrent web applications. And based on other tickets I'm not the only one.