KafkaProducerErrorHandling

KafkaProducer error handling

KafkaProducer tuple processing

For each input tuple one message is sent to Kafka for each configured topic using a standard consumer client api. During the process method the message is handed to the Kafka client, which is an asynchronous operation returning a Future. Tuple processing then continues with the operator returning from the process method.

The message is not delivered to Kafka until sometime later, and the Future can be used to determine when it has been received at the server (subject to the ack setting).

For consistent region the operator must ensure that at-least once processing is achieved, by ensuring that every message sent before the drain has been received at the server.

Error handling during tuple processing

The send message can fail with these exceptions:

• InterruptException - If the thread is interrupted while blocked
• SerializationException - If the key or value are not valid objects given the configured serializers
• TimeoutException - If the time taken for fetching metadata or allocating memory for the record has surpassed max.block.ms.
• KafkaException - If a Kafka related error occurs that does not belong to the public API exceptions

https://kafka.apache.org/0101/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html

These will be handled as:

• InterruptException - Rethrow error (or don't catch it) - operator is being shutdown.
• SerializationException - Rethrow error (or don't catch it) - Internal/application/data(?) error. Possible data error, for example an rstring with invalid UTF-8? Need to investigate.
• TimeoutException -
• KafkaException -

Note throwing an exception from the process method will result in that tuple being lost:

• consistent region - region will fail and tuples will be replayed from source. When a single tuple is sent to multiple topics, a message might have been sent to one or more of the topics before the failure.
• autonomous region - PE will fail and restart, no message has been sent for a single topic, when a single tuple is sent to multiple topics, a message might have been sent to one or more of the topics before the failure.