Databend Weekly Update Demo

[Xuanwo]

Overall Activity Summary

The Databend repository has seen significant activity over the past week, with contributions spanning new features, bug fixes, refactoring, and enhancements. Key themes include improvements to query optimization, support for advanced SQL features, and enhancements to the meta-service architecture. Several contributors have been actively involved in implementing new features and addressing critical bugs.

Key Ongoing Projects

1. Query Optimization Enhancements:

   • feat: impl Eliminate InSubquery #17628 by @KKould introduces a rule to eliminate simple subqueries, improving query performance.
   • feat: subquery to join case 0 #17617 by @KKould optimizes subqueries by converting them into joins, reducing redundant scans.
   • chore(optimizer): add tpcds unit test for optimizer #17605 by @BohuTANG adds TPC-DS unit tests for optimizer debugging.

2. Advanced SQL Features:

   • feat(query): add iceberg table functions #17626 by @sundy-li adds support for Iceberg table functions, enabling advanced analytics on Iceberg tables.
   • feat: UNPIVOT supports AS #17595 by @forsaken628 introduces UNPIVOT AS syntax, aligning Databend with Databricks-style SQL.

3. Meta-Service Refactoring:

   • refactor(meta): upgrade OpenRaft to v0.10.0-alpha.9 #17612 by @drmingdrmer upgrades OpenRaft to v0.10.0-alpha.9, enhancing the meta-service's underlying architecture.
   • refactor(meta): adopt external crate for shared state machine logic #17609 by @drmingdrmer adopts external crates for shared state machine logic, improving maintainability and modularity.

4. Spill and Memory Optimization:

   • refactor(query): reduce memory usage during aggregate spill #17550 by @zhang2014 reduces memory usage during aggregate spill, addressing performance bottlenecks.
   • fix: tpcds spill test fail #17597 by @forsaken628 fixes TPC-DS spill test failures and improves compatibility with cluster mode.

Priority Items

1. Critical Bug Fixes:

   • fix(query): fix cast nested types cause panic #17622 by @b41sh addresses a panic caused by casting nested types, ensuring stability in query execution.
   • fix(query): fix fold agg mismatch datatypes #17593 by @sundy-li fixes a mismatch in data types during aggregate folding, resolving a critical runtime error.
   • fix: the output columns of cte related columns lack the columns required by cte itself #17576 by @KKould resolves an issue with recursive CTEs, ensuring proper column handling.

2. Feature Completion:

   • feat: Add bendsave which can backup and restore databend data #17503 by @Xuanwo introduces bendsave, a disaster recovery tool for Databend. This feature is in progress and requires further testing and review.

3. Refactoring for Performance:

   • refactor: Improved Hilbert Clustering with Range Partition #17424 by @zhyass improves Hilbert clustering with range partitioning, reducing computational overhead.

Notable Discussions

• feat: strip_null_value func #17627: Proposal by @BohuTANG to introduce a new STRIP_NULL_VALUE() function for converting JSON nulls to SQL NULLs.
• feat: Support DEFAULT NEXTVAL for Sequences #17616: Discussion on supporting DEFAULT NEXTVAL for sequences, enabling easier sequence handling in table definitions.
• Feature: Support Displaying Runtime Filters in EXPLAIN Output #17608: Proposal by @zhang2014 to display runtime filters in EXPLAIN output, aiding query performance analysis.

Emerging Trends

1. Focus on Query Optimization: Multiple PRs and discussions indicate a strong emphasis on improving query performance through subquery elimination, join optimizations, and runtime filter enhancements.
2. Meta-Service Modularization: Refactoring efforts are consolidating shared logic into external crates, paving the way for a more modular and reusable architecture.
3. Iceberg Table Support: Growing interest in supporting Iceberg tables, as seen in feat(query): add iceberg table functions #17626 and feat(query): support create/drop table under iceberg catalog #17619, reflects a push towards advanced analytics capabilities.

Good First Issues

1. feat: CREATE OR REPLACE TASK syntax #17559: Add support for CREATE OR REPLACE TASK syntax. This issue involves extending existing DDL functionality and is a good entry point for contributors familiar with SQL parsing.
2. Feature: add sample directory with jupyter notebook #17460: Add a sample directory with Jupyter notebooks. This task is ideal for contributors interested in improving onboarding and documentation.
3. Tracking issues of outdated dependencies #16706: Address outdated dependencies. This involves upgrading specific dependencies and ensuring compatibility, making it a straightforward task for new contributors.
4. Feature: add isnan, isinf scalar functions #16690: Add isnan and isinf scalar functions for float types. This is a simple feature addition for contributors with basic Rust knowledge.