Skip to content

[FLINK-37834] Flink JDBC support ClickHouse Dialect #171

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

Biasqo
Copy link

@Biasqo Biasqo commented Jun 2, 2025

🔌 ClickHouse Dialect for Flink Link JDBC Connector

This project provides a ClickHouse SQL dialect implementation for the Apache Flink JDBC Connector.

🚀 Purpose
Apache Flink’s Link JDBC connector offers a flexible way to integrate with relational databases through dialects. However, native support for ClickHouse’s SQL features (e.g. merge trees, upserts, batch writes) is limited in SQL CLI.

✨ Features
✅ Custom ClickHouseDialect integration
✅ Column quoting and identifier case sensitivity fixes
✅ Insert/update query support tailored for ClickHouse (pkFields removed from update statement)

TODO:
Refactor Tests: Some methods are private (like getStreamFields() that's why I had to recreate some test classes)

Jira Issue:
https://issues.apache.org/jira/projects/FLINK/issues/FLINK-37834

@Biasqo
Copy link
Author

Biasqo commented Jun 7, 2025

@eskabetxe could you please start CI or review PR?

Copy link
Member

@eskabetxe eskabetxe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your contribution, I can't start CI/CD sorry. @1996fanrui can you start it?

I review the main code and left minor comments.
I dont review the tests as you have a TODO to some refactoring there.

You also need to add this module on flink-connector-jdbc-architecture.

.idea/vcs.xml Outdated
@@ -1,24 +1,24 @@
<?xml version="1.0" encoding="UTF-8"?>
<project version="4">
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the changes in this file should be reverted

<packaging>jar</packaging>

<properties>
<clickhouse.version>0.6.2</clickhouse.version>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not use the last version available 0.8.6?

@Biasqo
Copy link
Author

Biasqo commented Jun 12, 2025

@eskabetxe
Thanks for getting back to me

  1. I’m using version 0.6.2 since 0.8+ has some dependency issues with httpcore, and I didn’t want to change the existing httpcore version. If it’s okay to upgrade some of the libraries already included in the distribution, I can go ahead and fix it.
  2. I will revert .vcs
  3. About the tests: I’m not sure if I can override some private methods that are part of abstract test classes (for example, the abstract class TableBase, method getStreamFields). If I’m allowed to make them protected, the number of test classes could be significantly reduced. Clickhouse has its unique create statement - that's why it was a problem

@Biasqo
Copy link
Author

Biasqo commented Jun 14, 2025

Thanks for your contribution, I can't start CI/CD sorry. @1996fanrui can you start it?

I review the main code and left minor comments.

I dont review the tests as you have a TODO to some refactoring there.

You also need to add this module on flink-connector-jdbc-architecture.

Is it okay if I refactor some of the core classes in the tests, like making them protected?

@eskabetxe
Copy link
Member

@eskabetxe Thanks for getting back to me

  1. I’m using version 0.6.2 since 0.8+ has some dependency issues with httpcore, and I didn’t want to change the existing httpcore version. If it’s okay to upgrade some of the libraries already included in the distribution, I can go ahead and fix it.
  2. I will revert .vcs
  3. About the tests: I’m not sure if I can override some private methods that are part of abstract test classes (for example, the abstract class TableBase, method getStreamFields). If I’m allowed to make them protected, the number of test classes could be significantly reduced. Clickhouse has its unique create statement - that's why it was a problem

Its ok to bump some versions and is ok also change some test methods from private to protected if you need to override them..

Biasqo added 2 commits June 20, 2025 23:05
… archunit, 3. Driver version for tests = 0.6.2 (ClickHouse testcontainers support that version) 4. Refactored UPDATE statement
@Biasqo
Copy link
Author

Biasqo commented Jun 20, 2025

Ive refactored tests but cannot use click house driver version 0.7+ since click house testcontainers support 0.6.2.
Clickhouse driver dependency is only needed for tests

@Biasqo Biasqo requested a review from eskabetxe June 20, 2025 20:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants