Skip to content

Commit a30bdc3

Browse files
cnaurothdongjoon-hyun
authored andcommitted
[SPARK-51389][BUILD] Allow full customization of profiles in scalastyle
### What changes were proposed in this pull request? In the `scalastyle` helper script, activate profiles docker-integration-tests and kubernetes-integration-tests in the default value of `SPARK_PROFILES` instead of directly on the `sbt` command line. ### Why are the changes needed? The `scalastyle` script supports customization of which profiles to activate and check for style violations by passing command-line arguments that fill the `SPARK_PROFILES` environment variable. However, the docker-integration-tests and kubernetes-integration-tests profiles are always activated directly during the `sbt` invocation. This may waste time when engineers want to check for style violations on work in progress that doesn't change any code under these profiles. ### Does this PR introduce _any_ user-facing change? No. This change only impacts developer tooling. The default behavior of the script when no arguments are passed remains the same. ### How was this patch tested? In my local repo (not included in this pull request), I introduced style violations in docker-integration-tests and kubernetes-integration-tests. I then ran the script with no arguments, and it flagged the style violations: ``` dev/scalastyle Scalastyle checks failed at following occurrences: [error] /usr/local/google/home/cnauroth/git/apache/spark/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/Utils.scala:20:0: java.io. is in wrong order relative to java.nio.file.. [error] /usr/local/google/home/cnauroth/git/apache/spark/connector/docker-integration-tests/src/test/scala/org/apache/spark/util/DockerUtils.scala:23:0: scala.jdk.CollectionConverters._ is in wrong order relative to scala.sys.process._. [error] Total time: 66 s (01:06), completed Mar 4, 2025, 1:16:42 AM ``` I repeated this with command-line arguments specifying just the yarn profile, but it still flagged the violations in docker-integration-tests and kubernetes-integration-tests: ``` dev/scalastyle -Pyarn Scalastyle checks failed at following occurrences: [error] /usr/local/google/home/cnauroth/git/apache/spark/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/Utils.scala:20:0: java.io. is in wrong order relative to java.nio.file.. [error] /usr/local/google/home/cnauroth/git/apache/spark/connector/docker-integration-tests/src/test/scala/org/apache/spark/util/DockerUtils.scala:23:0: scala.jdk.CollectionConverters._ is in wrong order relative to scala.sys.process._. [error] Total time: 101 s (01:41), completed Mar 4, 2025, 9:04:29 PM ``` After applying the changes in the script, it still flags the violations in docker-integration-tests and kubernetes-integration-tests by default, but not when I specify only the yarn profile: ``` dev/scalastyle -Pyarn Scalastyle checks passed. ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #50156 from cnauroth/SPARK-51389. Authored-by: Chris Nauroth <cnauroth@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
1 parent cdba396 commit a30bdc3

File tree

1 file changed

+2
-3
lines changed

1 file changed

+2
-3
lines changed

dev/scalastyle

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -17,15 +17,13 @@
1717
# limitations under the License.
1818
#
1919

20-
SPARK_PROFILES=${1:-"-Pkubernetes -Pyarn -Pspark-ganglia-lgpl -Pkinesis-asl -Phive-thriftserver -Phive -Pvolcano -Pjvm-profiler -Phadoop-cloud"}
20+
SPARK_PROFILES=${1:-"-Pkubernetes -Pyarn -Pspark-ganglia-lgpl -Pkinesis-asl -Phive-thriftserver -Phive -Pvolcano -Pjvm-profiler -Phadoop-cloud -Pdocker-integration-tests -Pkubernetes-integration-tests"}
2121

2222
# NOTE: echo "q" is needed because SBT prompts the user for input on encountering a build file
2323
# with failure (either resolution or compilation); the "q" makes SBT quit.
2424
ERRORS=$(echo -e "q\n" \
2525
| build/sbt \
2626
${SPARK_PROFILES} \
27-
-Pdocker-integration-tests \
28-
-Pkubernetes-integration-tests \
2927
scalastyle test:scalastyle \
3028
| awk '{if($1~/error/)print}' \
3129
)
@@ -36,3 +34,4 @@ if test ! -z "$ERRORS"; then
3634
else
3735
echo -e "Scalastyle checks passed."
3836
fi
37+

0 commit comments

Comments
 (0)