[AMORO-3972][Core] Upgrade default Spark version from 3.3 to 3.5 (#3975)

* [AMORO-3972][Core] Upgrade default Spark version from 3.3 to 3.5

This PR upgrades the default Spark version from 3.3 to 3.5 in the Amoro project.

Changes:
- pom.xml: Update spark.version from 3.3.4 to 3.5.7 and spark.major.version from 3.3 to 3.5
- pom.xml (hadoop2 profile): Update spark.version from 3.3.4 to 3.5.7 and spark.major.version from 3.3 to 3.5
- docker/build.sh: Update SPARK_VERSION from 3.3.3 to 3.5.7
- docker/optimizer-spark/Dockerfile: Update ARG SPARK_VERSION from 3.3.3 to 3.5.7
- .github/workflows/docker-images.yml: Update Spark optimizer matrix from 3.3.3 to 3.5.7
- README.md: Update Spark optimizer default version documentation

Closes #3972

* Keep Spark 3.3 as default for hadoop2 profile

Address review feedback from @turboFei:
For hadoop2 profile, keep using the legacy Spark 3.3 by default
for better compatibility with Hadoop 2.x.

* Fix Spark version parameter name in README.md

Address review feedback from @turboFei:
Change -Dspark-optimizer.spark-version to -Dspark.version
This was missed in PR #3874.

Author: zhan7236

.github/workflows/docker-images.yml

@@ -186,7 +186,7 @@ jobs:
     if: ${{ startsWith(github.repository, 'apache/') }}
     strategy:
       matrix:
-        spark: [ "3.3.3" ]
+        spark: [ "3.5.7" ]
         scala: [ "2.12.15" ]
     steps:
       - uses: actions/checkout@v3
@@ -212,11 +212,11 @@ jobs:
           images: |
             name=apache/amoro-spark-optimizer
           tags: |
-            type=ref,event=branch,enable=${{ matrix.spark == '3.3.3' }},suffix=-snapshot
-            type=ref,event=branch,enable=${{ matrix.spark == '3.3.3' }},suffix=-snapshot-spark3.3
-            type=raw,enable=${{ matrix.hadoop == '3.3.3' && startsWith(github.ref, 'refs/tags/v') }},value=latest
-            type=semver,enable=${{ matrix.spark == '3.3.3' }},pattern={{version}}
-            type=semver,enable=${{ matrix.spark == '3.3.3' }},pattern={{version}}, suffix=-spark3.3
+            type=ref,event=branch,enable=${{ matrix.spark == '3.5.7' }},suffix=-snapshot
+            type=ref,event=branch,enable=${{ matrix.spark == '3.5.7' }},suffix=-snapshot-spark3.5
+            type=raw,enable=${{ matrix.hadoop == '3.5.7' && startsWith(github.ref, 'refs/tags/v') }},value=latest
+            type=semver,enable=${{ matrix.spark == '3.5.7' }},pattern={{version}}
+            type=semver,enable=${{ matrix.spark == '3.5.7' }},pattern={{version}}, suffix=-spark3.5
 
       - name: Print tags
         run: echo '${{ steps.meta.outputs.tags }}'

README.md

@@ -126,7 +126,7 @@ Amoro is built using Maven with JDK 8, 11 and 17(required for `amoro-format-mixe
 * Build with hadoop 2.x(the default is 3.x) dependencies: `./mvnw clean package -DskipTests -Phadoop2`
 * Specify Flink version for Flink optimizer(the default is 1.20.0): `./mvnw clean package -DskipTests -Dflink-optimizer.flink-version=1.20.0`
   * If the version of Flink is below 1.15.0, you also need to add the `-Pflink-optimizer-pre-1.15` parameter: `./mvnw clean package -DskipTests -Pflink-optimizer-pre-1.15 -Dflink-optimizer.flink-version=1.14.6`
-* Specify Spark version for Spark optimizer(the default is 3.3.3): `./mvnw clean package -DskipTests -Dspark-optimizer.spark-version=3.3.3`
+* Specify Spark version for Spark optimizer(the default is 3.5.7): `./mvnw clean package -DskipTests -Dspark.version=3.5.7`
 * Build `amoro-mixed-trino` module under JDK 17: `./mvnw clean package -DskipTests -Pformat-mixed-format-trino,build-mixed-format-trino -pl 'amoro-format-mixed/amoro-mixed-trino' -am`.
 * Build all modules: `./mvnw clean package -DskipTests -Ptoolchain,build-mixed-format-trino`, besides you need config `toolchains.xml` in `${user.home}/.m2/` dir with content below.
 * Build a distribution package with all formats integrated: `./mvnw clean package -Psupport-all-formats`

docker/build.sh

@@ -27,7 +27,7 @@ cd $CURRENT_DIR
 
 AMORO_VERSION=`cat $PROJECT_HOME/pom.xml | grep 'amoro-parent' -C 3 | grep -Eo '<version>.*</version>' | awk -F'[><]' '{print $3}'`
 FLINK_VERSION=1.20.0
-SPARK_VERSION=3.3.3
+SPARK_VERSION=3.5.7
 DEBIAN_MIRROR=http://deb.debian.org
 APACHE_ARCHIVE=https://archive.apache.org/dist
 FLINK_OPTIMIZER_JOB_PATH=amoro-optimizer/amoro-optimizer-flink/target/amoro-optimizer-flink-${AMORO_VERSION}-jar-with-dependencies.jar
@@ -50,7 +50,7 @@ Images:
 
 Options:
     --flink-version         Flink binary release version, default is 1.20.0, format must be x.y.z
-    --spark-version         Spark binary release version, default is 3.3.3, format must be x.y.z
+    --spark-version         Spark binary release version, default is 3.5.7, format must be x.y.z
     --apache-archive        Apache Archive url, default is https://archive.apache.org/dist
     --debian-mirror         Mirror url of debian, default is http://deb.debian.org
     --maven-mirror          Mirror url of maven, default is https://repo.maven.apache.org/maven2

docker/optimizer-spark/Dockerfile

@@ -14,7 +14,7 @@
 #  See the License for the specific language governing permissions and
 # limitations under the License.
 
-ARG SPARK_VERSION=3.3.3
+ARG SPARK_VERSION=3.5.7
 
 FROM apache/spark:${SPARK_VERSION}
 

pom.xml

@@ -126,8 +126,8 @@
         <orc-core.version>1.9.7</orc-core.version>
         <awssdk.version>2.24.12</awssdk.version>
         <aliyun-sdk-oss.version>3.10.2</aliyun-sdk-oss.version>
-        <spark.version>3.3.4</spark.version>
-        <spark.major.version>3.3</spark.major.version>
+        <spark.version>3.5.7</spark.version>
+        <spark.major.version>3.5</spark.major.version>
         <dropwizard.metrics.version>4.2.19</dropwizard.metrics.version>
         <caffeine.version>2.9.3</caffeine.version>
         <snakeyaml.version>2.2</snakeyaml.version>