TEST: enable pushdown_filters and reorder_filters by default

[alamb]

( I am using this PR to test, I don't intend to merge it yet )

Which issue does this PR close?

• Closes Enable parquet filter pushdown (filter_pushdown) by default #3463

Rationale for this change

We have made non trivial progress in filter representation in Parquet. Let's see where performance is now.

What changes are included in this PR?

• base on [WIP] Update to arrow, parquet 57.1.0 #18820
• Enable pushdown_filters and reorder_filters

Are these changes tested?

By CI tests

Are there any user-facing changes?

[alamb]

🤖 ./gh_compare_branch.sh Benchmark Script Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/pushdown_filters_test (8a06d63) to d65fb86 diff using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete

[alamb]

🤖: Benchmark completed

Details

Comparing HEAD and alamb_pushdown_filters_test
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ alamb_pushdown_filters_test ┃         Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
│ QQuery 0     │  2704.76 ms │                  2693.83 ms │      no change │
│ QQuery 1     │  1211.14 ms │                  1278.46 ms │   1.06x slower │
│ QQuery 2     │  2313.54 ms │                  2408.79 ms │      no change │
│ QQuery 3     │  1159.95 ms │                  1156.61 ms │      no change │
│ QQuery 4     │  2336.83 ms │                  2351.37 ms │      no change │
│ QQuery 5     │ 28195.81 ms │                 27958.12 ms │      no change │
│ QQuery 6     │  4058.62 ms │                   108.14 ms │ +37.53x faster │
│ QQuery 7     │  3742.15 ms │                  3701.67 ms │      no change │
└──────────────┴─────────────┴─────────────────────────────┴────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                          ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                          │ 45722.81ms │
│ Total Time (alamb_pushdown_filters_test)   │ 41656.99ms │
│ Average Time (HEAD)                        │  5715.35ms │
│ Average Time (alamb_pushdown_filters_test) │  5207.12ms │
│ Queries Faster                             │          1 │
│ Queries Slower                             │          1 │
│ Queries with No Change                     │          6 │
│ Queries with Failure                       │          0 │
└────────────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ alamb_pushdown_filters_test ┃         Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
│ QQuery 0     │     2.34 ms │                     2.61 ms │   1.12x slower │
│ QQuery 1     │    50.62 ms │                    51.87 ms │      no change │
│ QQuery 2     │   136.14 ms │                   136.47 ms │      no change │
│ QQuery 3     │   162.09 ms │                   160.21 ms │      no change │
│ QQuery 4     │  1098.61 ms │                  1095.76 ms │      no change │
│ QQuery 5     │  1477.67 ms │                  1491.19 ms │      no change │
│ QQuery 6     │     2.17 ms │                     2.16 ms │      no change │
│ QQuery 7     │    54.19 ms │                    68.48 ms │   1.26x slower │
│ QQuery 8     │  1444.55 ms │                  1422.06 ms │      no change │
│ QQuery 9     │  1890.35 ms │                  1852.51 ms │      no change │
│ QQuery 10    │   363.24 ms │                   484.80 ms │   1.33x slower │
│ QQuery 11    │   427.02 ms │                   532.87 ms │   1.25x slower │
│ QQuery 12    │  1340.34 ms │                  1545.03 ms │   1.15x slower │
│ QQuery 13    │  2077.33 ms │                  2280.93 ms │   1.10x slower │
│ QQuery 14    │  1274.21 ms │                  1448.68 ms │   1.14x slower │
│ QQuery 15    │  1277.86 ms │                  1261.35 ms │      no change │
│ QQuery 16    │  2677.07 ms │                  2704.20 ms │      no change │
│ QQuery 17    │  2662.55 ms │                  2692.03 ms │      no change │
│ QQuery 18    │  5448.25 ms │                  4951.08 ms │  +1.10x faster │
│ QQuery 19    │   130.47 ms │                   143.08 ms │   1.10x slower │
│ QQuery 20    │  2003.32 ms │                  1854.73 ms │  +1.08x faster │
│ QQuery 21    │  2355.36 ms │                  2330.38 ms │      no change │
│ QQuery 22    │  3991.87 ms │                  3944.03 ms │      no change │
│ QQuery 23    │ 16491.78 ms │                  1080.71 ms │ +15.26x faster │
│ QQuery 24    │   215.96 ms │                   253.74 ms │   1.17x slower │
│ QQuery 25    │   489.11 ms │                   617.52 ms │   1.26x slower │
│ QQuery 26    │   228.11 ms │                   324.39 ms │   1.42x slower │
│ QQuery 27    │  2868.16 ms │                  2929.86 ms │      no change │
│ QQuery 28    │ 23345.61 ms │                 23660.66 ms │      no change │
│ QQuery 29    │   975.88 ms │                   994.78 ms │      no change │
│ QQuery 30    │  1333.55 ms │                  1354.66 ms │      no change │
│ QQuery 31    │  1416.85 ms │                  1351.01 ms │      no change │
│ QQuery 32    │  5442.42 ms │                  4533.41 ms │  +1.20x faster │
│ QQuery 33    │  6011.79 ms │                  5754.30 ms │      no change │
│ QQuery 34    │  6139.89 ms │                  5858.52 ms │      no change │
│ QQuery 35    │  1950.34 ms │                  1865.06 ms │      no change │
│ QQuery 36    │   121.36 ms │                    25.69 ms │  +4.72x faster │
│ QQuery 37    │    51.21 ms │                    25.18 ms │  +2.03x faster │
│ QQuery 38    │   120.24 ms │                    25.57 ms │  +4.70x faster │
│ QQuery 39    │   194.03 ms │                    25.93 ms │  +7.48x faster │
│ QQuery 40    │    43.31 ms │                    26.00 ms │  +1.67x faster │
│ QQuery 41    │    40.36 ms │                    25.77 ms │  +1.57x faster │
│ QQuery 42    │    31.97 ms │                    25.25 ms │  +1.27x faster │
└──────────────┴─────────────┴─────────────────────────────┴────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                          ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                          │ 99859.52ms │
│ Total Time (alamb_pushdown_filters_test)   │ 83214.54ms │
│ Average Time (HEAD)                        │  2322.31ms │
│ Average Time (alamb_pushdown_filters_test) │  1935.22ms │
│ Queries Faster                             │         11 │
│ Queries Slower                             │         11 │
│ Queries with No Change                     │         21 │
│ Queries with Failure                       │          0 │
└────────────────────────────────────────────┴────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃      HEAD ┃ alamb_pushdown_filters_test ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1     │ 131.35 ms │                   139.34 ms │  1.06x slower │
│ QQuery 2     │  29.56 ms │                    28.82 ms │     no change │
│ QQuery 3     │  34.98 ms │                    39.69 ms │  1.13x slower │
│ QQuery 4     │  29.36 ms │                    29.42 ms │     no change │
│ QQuery 5     │  87.73 ms │                    88.77 ms │     no change │
│ QQuery 6     │  19.59 ms │                    19.82 ms │     no change │
│ QQuery 7     │ 217.18 ms │                   221.16 ms │     no change │
│ QQuery 8     │  32.12 ms │                    34.75 ms │  1.08x slower │
│ QQuery 9     │ 103.41 ms │                   107.77 ms │     no change │
│ QQuery 10    │  64.14 ms │                    64.51 ms │     no change │
│ QQuery 11    │  19.10 ms │                    17.62 ms │ +1.08x faster │
│ QQuery 12    │  52.10 ms │                    51.44 ms │     no change │
│ QQuery 13    │  47.55 ms │                    46.22 ms │     no change │
│ QQuery 14    │  13.94 ms │                    13.57 ms │     no change │
│ QQuery 15    │  25.01 ms │                    24.55 ms │     no change │
│ QQuery 16    │  25.81 ms │                    25.46 ms │     no change │
│ QQuery 17    │ 148.10 ms │                   148.29 ms │     no change │
│ QQuery 18    │ 282.55 ms │                   280.77 ms │     no change │
│ QQuery 19    │  49.21 ms │                    38.45 ms │ +1.28x faster │
│ QQuery 20    │  50.67 ms │                    49.56 ms │     no change │
│ QQuery 21    │ 338.10 ms │                   320.93 ms │ +1.05x faster │
│ QQuery 22    │  17.73 ms │                    17.73 ms │     no change │
└──────────────┴───────────┴─────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary                          ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)                          │ 1819.30ms │
│ Total Time (alamb_pushdown_filters_test)   │ 1808.63ms │
│ Average Time (HEAD)                        │   82.70ms │
│ Average Time (alamb_pushdown_filters_test) │   82.21ms │
│ Queries Faster                             │         3 │
│ Queries Slower                             │         3 │
│ Queries with No Change                     │        16 │
│ Queries with Failure                       │         0 │
└────────────────────────────────────────────┴───────────┘

[alamb]

I am also testing with just filter_pushdown on:

• TEST: enable pushdown_filters by default #18874

I am going to focus my efforts on profiling these queries which seem to have gotten the most slower:

│ QQuery 24    │   215.96 ms │                   253.74 ms │   1.17x slower │
│ QQuery 25    │   489.11 ms │                   617.52 ms │   1.26x slower │
│ QQuery 26    │   228.11 ms │                   324.39 ms │   1.42x slower │

Here is the query:

set datafusion.execution.parquet.binary_as_string = true
SELECT "SearchPhrase" FROM hits WHERE "SearchPhrase" <> '' ORDER BY "SearchPhrase" LIMIT 10;

Basically my next steps are to profile these queries and see what is slower (and if it is related to filter representation, I will go focus on apache/arrow-rs#8902)

[Dandandan]

I am also testing with just filter_pushdown on:

• TEST: enable pushdown_filters by default #18874

I am going to focus my efforts on profiling these queries which seem to have gotten the most slower:

│ QQuery 24    │   215.96 ms │                   253.74 ms │   1.17x slower │
│ QQuery 25    │   489.11 ms │                   617.52 ms │   1.26x slower │
│ QQuery 26    │   228.11 ms │                   324.39 ms │   1.42x slower │

Here is the query:

set datafusion.execution.parquet.binary_as_string = true
SELECT "SearchPhrase" FROM hits WHERE "SearchPhrase" <> '' ORDER BY "SearchPhrase" LIMIT 10;

Basically my next steps are to profile these queries and see what is slower (and if it is related to filter representation, I will go focus on apache/arrow-rs#8902)

Looks like we are very close!

FYI, there a couple more slow than query 24:

│ QQuery 26    │   228.11 ms │                   324.39 ms │   1.42x slower │
│ QQuery 10    │   363.24 ms │                   484.80 ms │   1.33x slower │
│ QQuery 7     │    54.19 ms │                    68.48 ms │   1.26x slower │
│ QQuery 25    │   489.11 ms │                   617.52 ms │   1.26x slower │
│ QQuery 11    │   427.02 ms │                   532.87 ms │   1.25x slower │