Hi all, please forgive me if I've misunderstood the current behaviour.
Afaik when configured batch/manual mode if an acknowledge isn't called before the next batch starts the only recourse is to call consumer.commitSync(..) ?
For my usecase I want to be able to read and buffer across many polls/batches without any ack until eventually some condition is met at which point some processing is done on the buffered messages and eventually the last one acked.
I also want to be able to throw errors during the processing (or at any point) and have the retry policy and dlq handling work.
This almost works out of the box, and I think most of the machinery is there already to support this.
I'd like to see a variant of the Acknowledge interface/impl that accepts a ConsumerRecord instead of just the batch index option and have the listener container use the offset and partition directly to ack and not care about the current batch. it seems to me this is almost whats happening anyway but there seem to be a lot of gates that care about whether the ack relates to the current batch or not.
With Exceptions, the existing BatchListenerFailedException already accepts a ConsumerRecord and could bypass the existing gates that check if its part of the current batch and invoke the retry/dlq handling already configured. if retry of the offending record(s) (we could also pass a list of records to BatchListenerFailedException and they could be used as the retry set) fails then invoke the errorhandler and seek to make behaviour consistent with what happens when an error in batch occurs.
This would allow truly async handling without everyone rolling their own half baked workarounds whilst allowing a clean, non blocking main thread to maintain a happy state with the broker and supporting usecases like this.
Have I missed something? Is there already a means to handle this?
thoughts?
Thankyou.
Hi all, please forgive me if I've misunderstood the current behaviour.
Afaik when configured batch/manual mode if an acknowledge isn't called before the next batch starts the only recourse is to call consumer.commitSync(..) ?
For my usecase I want to be able to read and buffer across many polls/batches without any ack until eventually some condition is met at which point some processing is done on the buffered messages and eventually the last one acked.
I also want to be able to throw errors during the processing (or at any point) and have the retry policy and dlq handling work.
This almost works out of the box, and I think most of the machinery is there already to support this.
I'd like to see a variant of the Acknowledge interface/impl that accepts a ConsumerRecord instead of just the batch index option and have the listener container use the offset and partition directly to ack and not care about the current batch. it seems to me this is almost whats happening anyway but there seem to be a lot of gates that care about whether the ack relates to the current batch or not.
With Exceptions, the existing BatchListenerFailedException already accepts a ConsumerRecord and could bypass the existing gates that check if its part of the current batch and invoke the retry/dlq handling already configured. if retry of the offending record(s) (we could also pass a list of records to BatchListenerFailedException and they could be used as the retry set) fails then invoke the errorhandler and seek to make behaviour consistent with what happens when an error in batch occurs.
This would allow truly async handling without everyone rolling their own half baked workarounds whilst allowing a clean, non blocking main thread to maintain a happy state with the broker and supporting usecases like this.
Have I missed something? Is there already a means to handle this?
thoughts?
Thankyou.