Track messages that successfully completed the message or error pipeline but failed to get acknowledged due to expired leases in receiveonly mode

[danielmarbach]

Related issue:

• ASB transport using receiveOnly mode cannot forward messages to the error queue when handler execution exceeds message lock duration #1043

This PR follows through on related changes done in Amazon SQS but adjusts the core pieces of the change to Azure Service Bus-specific behavior. Additionally, it brings in a number of layered tests that verify the code works as expected and doesn't accidentally break later.

An LRU cache to track the delivery attempts is not necessary because the receive count is automatically updated by the storage queue infrastructure.

The changes have only been applied for the receive only transaction mode because with the sends-atomic receive mode the outgoing messages are enlisted with a send-via information together with the incoming message. The outgoing messages are only sent when the incoming message successfully completed. In cases when the incoming message wasn't successfully completed, the outgoing operations are discarded by the broker and there is no way to refer to outgoing operations of a previous message handling attempt.

• Caching Acknowledged Messages: A cache (using BitFaster.Caching.Lru) is introduced to keep track of messages that have been marked as successfully completed. This helps in avoiding unnecessary retries for messages that are already acknowledged.
• Error Handling Improvements: Enhanced handling around retries and error recovery, including better logging and recovery actions for different failure scenarios.
• Minor cleanups such as removing the async suffix or making the abandon actually safe.

Before ReceiveOnly

graph TD
    A[Receive Message] --> B[Invoke OnMessage]
    B --> C{Success?}
    C -- Yes --> D[Acknowledge message]
    D --> E[Message acknowledged]  
    C -- No --> F[Invoke OnError]
    F --> G{Error requires retry?}
    G -- Yes --> H[Requeue message]
    G -- No --> I[Acknowledge message]
    H --> J[Message requeued]
    I --> E[Message acknowledged]

Loading

After ReceiveOnly

graph TD
    A[Receive Message] --> B[ID in LRU]
    B -- Yes --> M[Acknowledge message]
    F --> G[Message tracked]
    B -- No --> H[Invoke OnMessage]
    H --> I{Success?}
    I -- Yes --> M
    I -- No --> J[Invoke OnError]
    J --> K{Error requires retry?}
    K -- Yes --> L[Requeue message]
    K -- No --> M[Acknowledge message]
    M --> N{Ack success?}
    N -- Yes --> E[Message acknowledged]
    N -- No --> F[Track ID in LRU]
    G -- Message requeued --> A

Loading

Tradeoffs

The tradeoffs are the sames as the other fixes applied in RabbitMQ and Amazon SQS. In competing consumer scenarios there can still be more retries because the LRU cache is in memory, but it is still marginally better than having a potentially unlimited number of retries.

As for the transaction mode None, there is no change required since we are not giving any guarantees there and messages are acknowledged early on with ReceiveAndDelete.

Ideally at some point, if the complexity grows further, it would be good to split the pump into different receive strategies. Given the message settlement operations already had specific code around transport transaction modes and the risk of doing such a split is high a decision was made to not introduce a receive strategy and treat the settlement operation extensions as closesly related to the pump (hence they share the same logger)

[danielmarbach]

There is intermittent flakiness happening with the concurrency tests that got introduced recently. It also happens on master so it is quite likely not related to the changes proposed in this PR

[danielmarbach]

Requires a rebase after #1042 has been merged