You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[SPARK-49507][SQL] Fix the case issue after enabling metastorePartitionPruningFastFallback
### What changes were proposed in this pull request?
This PR enhances the partition predicate handling in the `HiveShim` by ensuring that partition schema and predicates are properly transformed to lowercase. This change improves compatibility when generating partition predicates for filtering.
How to reproduce:
```
CREATE TABLE t (ID BIGINT, DT STRING) USING parquet PARTITIONED BY (DT);
set spark.sql.hive.metastorePartitionPruningFastFallback=true;
select * from t where dt=20240820;
```
Error message:
```
org.apache.spark.sql.AnalysisException: Expected only partition pruning predicates: List(isnotnull(DT#21), (cast(DT#21 as bigint) = 20240820)).
at org.apache.spark.sql.errors.QueryCompilationErrors$.nonPartitionPruningPredicatesNotExpectedError(QueryCompilationErrors.scala:2414)
at org.apache.spark.sql.catalyst.catalog.ExternalCatalogUtils$.generatePartitionPredicateByFilter(ExternalCatalo
```
### Why are the changes needed?
Bug fix.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Unit test.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#47998 from TongWei1105/SPARK-49507.
Authored-by: TongWei1105 <vvtwow@gmail.com>
Signed-off-by: Yuming Wang <yumwang@ebay.com>
0 commit comments