Skip to content

Deserialization failure handling #15

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed

Conversation

ferenc-csaky
Copy link
Contributor

@ferenc-csaky ferenc-csaky commented Feb 20, 2025

This is a work in progress thing, just wanted to share ASAP for everyone to get the idea.

Relevant changes

  • Modified classes excluded from the connector JAR.
  • Copied the classes that has to be modified.
    • KafkaDynamicTableFactory defines the new table options (scan.deser.failure-handler, scan.deser.failure-topic) and passes their value.
    • KafkaDynamicSource in this context is just a connecting layer that passes through the newly added information.
    • DynamicKafkaDeserializationSchema is modified, a newly added deserWithFailureHandling method now can handle the exceptions accordingly.
  • Added a skeleton logic for failure handling.

TODO

  • The deserFailureTarget is now an empty shell, which should be a KafkaProducer, that has to be initialized accordingly.
  • The same has to be done for the upsert-kafka classes too.
  • Do some tests.

@ferenc-csaky ferenc-csaky force-pushed the deser-failure-handler branch 2 times, most recently from 24dabd4 to 5dad322 Compare March 14, 2025 18:56
@ferenc-csaky
Copy link
Contributor Author

@velo Would it be possible to skip the enforced code format for the classes that was duplicated from the flink-connector-kafka repo? I believe that code should be kept as is cause if it has to be updated from a more recent connector version, it would be much easier to track what actually changed if the format is kept as is.

@ferenc-csaky ferenc-csaky force-pushed the deser-failure-handler branch from 5dad322 to e855980 Compare April 1, 2025 15:55
@ferenc-csaky ferenc-csaky marked this pull request as ready for review April 1, 2025 15:58
@velo velo force-pushed the main branch 3 times, most recently from 1880d8c to 60c9755 Compare April 3, 2025 14:39
@ferenc-csaky
Copy link
Contributor Author

Superseded by #68

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant