You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[SPARK-51389][BUILD] Allow full customization of profiles in scalastyle
### What changes were proposed in this pull request?
In the `scalastyle` helper script, activate profiles docker-integration-tests and kubernetes-integration-tests in the default value of `SPARK_PROFILES` instead of directly on the `sbt` command line.
### Why are the changes needed?
The `scalastyle` script supports customization of which profiles to activate and check for style violations by passing command-line arguments that fill the `SPARK_PROFILES` environment variable. However, the docker-integration-tests and kubernetes-integration-tests profiles are always activated directly during the `sbt` invocation. This may waste time when engineers want to check for style violations on work in progress that doesn't change any code under these profiles.
### Does this PR introduce _any_ user-facing change?
No. This change only impacts developer tooling. The default behavior of the script when no arguments are passed remains the same.
### How was this patch tested?
In my local repo (not included in this pull request), I introduced style violations in docker-integration-tests and kubernetes-integration-tests. I then ran the script with no arguments, and it flagged the style violations:
```
dev/scalastyle
Scalastyle checks failed at following occurrences:
[error] /usr/local/google/home/cnauroth/git/apache/spark/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/Utils.scala:20:0: java.io. is in wrong order relative to java.nio.file..
[error] /usr/local/google/home/cnauroth/git/apache/spark/connector/docker-integration-tests/src/test/scala/org/apache/spark/util/DockerUtils.scala:23:0: scala.jdk.CollectionConverters._ is in wrong order relative to scala.sys.process._.
[error] Total time: 66 s (01:06), completed Mar 4, 2025, 1:16:42 AM
```
I repeated this with command-line arguments specifying just the yarn profile, but it still flagged the violations in docker-integration-tests and kubernetes-integration-tests:
```
dev/scalastyle -Pyarn
Scalastyle checks failed at following occurrences:
[error] /usr/local/google/home/cnauroth/git/apache/spark/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/Utils.scala:20:0: java.io. is in wrong order relative to java.nio.file..
[error] /usr/local/google/home/cnauroth/git/apache/spark/connector/docker-integration-tests/src/test/scala/org/apache/spark/util/DockerUtils.scala:23:0: scala.jdk.CollectionConverters._ is in wrong order relative to scala.sys.process._.
[error] Total time: 101 s (01:41), completed Mar 4, 2025, 9:04:29 PM
```
After applying the changes in the script, it still flags the violations in docker-integration-tests and kubernetes-integration-tests by default, but not when I specify only the yarn profile:
```
dev/scalastyle -Pyarn
Scalastyle checks passed.
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#50156 from cnauroth/SPARK-51389.
Authored-by: Chris Nauroth <cnauroth@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
0 commit comments