Skip to content

Propose:Use OpenDAL HdfsNative Replace current HDFS #17734

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wubx opened this issue Apr 8, 2025 · 0 comments
Open

Propose:Use OpenDAL HdfsNative Replace current HDFS #17734

wubx opened this issue Apr 8, 2025 · 0 comments

Comments

@wubx
Copy link
Member

wubx commented Apr 8, 2025

Summary
We are currently utilizing Databend to create a stage using HDFS, as detailed in the Databend documentation. This implementation requires compiling Databend with Java and providing full support for the Hadoop package, which involves a highly complex configuration.

The troubleshooting process for any errors encountered during this setup has proven to be extremely intricate.

To simplify our operations, I propose replacing the current HDFS implementation with OpenDAL HDFS Native. This solution can streamline our processes, as outlined in the following resources:

OpenDAL HDFS Native Documentation
OpenDAL HDFS Native GitHub Repository

Adopting OpenDAL HDFS Native could reduce complexity and improve our efficiency in working with HDFS in the Databend | Databend Cloud environment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant