[DISCUSS] We have some C++ code to share for Iceberg read/write

[jovezhong]

Just a quick note, we are open-sourcing our C++ implementation for Iceberg read/write, starting from this "small" PR: timeplus-io/proton#928

It supports Iceberg REST Catalog (verified with AWS Glue, WIP for Polaris and Gravitino). Primarily for S3 storage (half-done with new S3 table). Streaming writes and batch read.

We leveraged some code from ClickHouse. Happy to work with you to contribute in one way or the other.

[jovezhong]

Quick update: S3 table supported is added in preview build 3, as well as validation with Apache Gravitino. Check more info in the PR

[lidavidm]

Thanks! It would be interesting to find a way to integrate this here

[jovezhong]

Sure, loop in @zliang-min who works on the actual implementation from our team. We found a lot of tricky issues in Glue/Athena/Spark/S3Table. For example, some data written by Timeplus can be read by SparkSQL but not Athena. Maybe we can get some help from you guys or exchange some lessons&learns. If there is anything specific we can help, let us know.

ICEBERG_CANNOT_OPEN_SPLIT: Error opening Iceberg split s3://tp-iceberg-demo/tripdata_spark/data/00000-0-c8222b82-5104-48ee-a637-e2fd608736ee-0-00001.parquet (offset=201326592, length=67108864): Read 49152 tail bytes of file io.trino.filesystem.hdfs.aws.AthenaHdfsInputFile@745c9779 failed: Incorrect file size (524571179) for file (end of stream not reached): io.trino.filesystem.hdfs.aws.AthenaHdfsInputFile@745c9779

[lidavidm]

Also @wgtmac any thoughts? It will probably be some time before we're prepared to tackle this though (iceberg-cpp is still sketching out the core APIs)

[wgtmac]

I agree with @lidavidm. We need to finalize core APIs (especially on how to integrate with different engines and I/O approaches) before concrete implementations.