Simplify test_dependency_versions.sh (#1758)

Removes almost all of the copied `deps/test` sources and targets in
favor of invoking `@rules_scala` and `@multi_frameworks_toolchain` tests
directly. Moves targets depending on `@rules_python` and `@rules_shell`
to new packages so test packages don't break when `@rules_scala` isn't
the main module.

Introduces `RULES_SCALA_TARGETS`, `MULTI_FRAMEWORKS_TOOLCHAIN_TARGETS`,
and `ALL_TARGETS` arrays to easily specify compatibility test targets
instead of copying them. Only keeps a single `ScalafmtTest` target
within `deps/test/BUILD.bazel.test`, which is a special case (described
below) and executes successfully on Windows.

Though these are new packages, they're comprised of existing files and
targets from their parent packages:

- //test/jmh/runtime
- //test/sh_tests
- //test/src/main/scala/scalarules/test/twitter_scrooge/strings

Also:

- Marks every `bazel_dep` for `@latest_dependencies` with
  `dev_dependency = True`.

- Replaces `$(location ...)` instances in moved targets with
  `$(rootpath ...)` (and in one case, with `$(execpath ...)`) per
  bazelbuild/bazel#25198.

---

While drafting a blog post describing `test_dependency_versions.sh`, I
revisited the decision to copy targets and files in #1729 and #1738.
Executing test targets from `@rules_scala` directly from the test module
actually works, making the test far less complex, and making for better
testing advice.

This required moving targets that referenced dev dependencies to their
own packages, fixing `@rules_scala` test package breakages. Making the
`@latest_dependencies` module a dev dependency eliminated module version
warnings from `@multi_frameworks_toolchain`, since the test sets older
dependency versions. Then making `@latest_dependencies` a dev dependency
everywhere, instead of only in `@multi_frameworks_toolchain`, seemed
more consistent.

The test retains the `//:ScalafmtTest` target because `rules_scala`
doesn't actually provide Scalafmt rules, but only provides
`ext_scalafmt` to create custom rules, per:

- docs/phase_scalafmt.md
- docs/customizable_phase.md

I discovered this after going down a rabbit hole regarding the
implicitly defined (and ostensibly deprecated) `.format` and
`.format-test` predeclared outputs for Scalafmt rules. Long story short,
implicitly defined predeclared outputs have been "deprecated" since
2018-03-05, predating Bazel 0.28.0:

- https://bazel.build/versions/8.3.0/extending/rules#requesting_output_files
- https://bazel.build/versions/8.3.0/extending/rules#deprecated_predeclared_outputs
- https://bazel.build/versions/8.3.0/rules/lib/globals/bzl#rule.outputs
- 2018-03-05: Output Map Madness
  https://docs.google.com/document/d/1ic9lJPn-0VqgKcqSbclVWwYDW2eiV-9k6ZUK_xE6H5E/edit
- 2019-04-08: bazelbuild/bazel#7977: incompatible_no_rule_outputs_param:
  disable outputs param of rule function
  bazelbuild/bazel#7977
- 2019-06-07: Rollforward "Disable outputs param of rule function" with
  fix (introduced --incompatible_no_rule_outputs_param in Bazel 0.28.0)
  bazelbuild/bazel@36c70a6
- 2019-09-04: Document the replacements for the `outputs` parameter to
  the `rule()` function. (Bazel 1.0.0)
  bazelbuild/bazel@e29ddda

However, I learned that the `.format` and `.format-test` targets are
Bash scripts run _after_ the original rule executes Scalafmt. The
generated scripts don't depend on the `//scala/scalafmt:scalafmt` binary
at all. Plus, they only work when invoked within the main module, since
they reference relative paths within `BUILD_WORKSPACE_DIRECTORY`. And
finally, `bazel test @rules_scala//test/scalafmt:formatted-test` fails
because it tries to declare output files in a nonexistent directory.

```txt
ERROR: .../external/rules_scala~/test/scalafmt/BUILD:43:20:
  in scalafmt_scala_test rule
  @@rules_scala~//test/scalafmt:formatted-test:

Traceback (most recent call last):
  File ".../external/rules_scala~/scala/private/rules/scala_test.bzl",
    line 38, column 22, in _scala_test_impl
      return run_phases(
  File ".../external/rules_scala~/scala/private/phases/api.bzl",
    line 45, column 23, in run_phases
      return _run_phases(ctx, builtin_customizable_phases, target = None)
  File ".../external/rules_scala~/scala/private/phases/api.bzl",
    line 77, column 32, in _run_phases
      new_provider = function(ctx, current_provider)
  File ".../external/rules_scala~/scala/private/phases/phase_scalafmt.bzl",
    line 10, column 46, in phase_scalafmt
      manifest, files, srcs = _build_format(ctx)
  File ".../external/rules_scala~/scala/private/phases/phase_scalafmt.bzl",
    line 30, column 44, in _build_format
      file = ctx.actions.declare_file("{}.fmt.output".format(src.short_path))

Error in declare_file:
  the output artifact
  'external/rules_scala~/test/rules_scala~/test/scalafmt/formatted/formatted-test.scala.fmt.output'
  is not under package directory
  'external/rules_scala~/test/scalafmt'
  for target '@@rules_scala~//test/scalafmt:formatted-test'

ERROR: .../external/rules_scala~/test/scalafmt/BUILD:43:20:
  Analysis of target '@@rules_scala~//test/scalafmt:formatted-test'
  failed
```

So invoking `bazel test //:ScalafmtTest` (via `/...` from `ALL_TARGETS`)
is sufficient, since we're only validating that the toolchains execute
properly. (`test_scalafmt.sh` tests the behavior of the `.format` and
`.format-test` scripts, _including_ on Windows via `--run_under=bash`.)
This also means we can invoke this target on Windows, instead of having
special case logic to avoid invoking the previous `bazel run` command.

Author: mbland

deps/test/BUILD.bazel.test

@@ -1,156 +1,11 @@
-"""Test targets to ensure dependency version compatibility.
+"""Test targets to ensure dependency version compatibility."""
 
-Copied and adapted targets from the main repo as noted.
-"""
-load(
-    ":defs.bzl",
-    "default_outputs_test",
-    "scalafmt_scala_test",
-    "scrooge_transitive_outputs_test",
-)
-load("@rules_java//java:defs.bzl", "java_library")
-load("@rules_proto//proto:defs.bzl", "proto_library")
-load("@rules_scala//jmh:jmh.bzl", "scala_benchmark_jmh")
-load("@rules_scala//scala/scalafmt:phase_scalafmt_ext.bzl", "ext_scalafmt")
-load("@rules_scala//scala:advanced_usage/scala.bzl", "make_scala_test")
-load(
-    "@rules_scala//scala:scala.bzl",
-    "scala_binary",
-    "scala_doc",
-    "scala_junit_test",
-    "scala_library",
-    "scala_specs2_junit_test",
-    "scala_test",
-)
-load("@rules_scala//scala_proto:scala_proto.bzl", "scala_proto_library")
-load("@rules_scala//thrift:thrift.bzl", "thrift_library")
-load(
-    "@rules_scala//twitter_scrooge:twitter_scrooge.bzl",
-    "scrooge_java_library",
-    "scrooge_scala_library",
-)
-
-# From: `test/BUILD`
-scala_binary(
-    name = "ScalaBinary",
-    srcs = ["ScalaBinary.scala"],
-    main_class = "scalarules.test.ScalaBinary",
-    deps = [
-        ":HelloLib",
-    ],
-)
-
-scala_library(
-    name = "HelloLib",
-    srcs = ["HelloLib.scala"],
-)
-
-scala_doc(
-    name = "ScalaDoc",
-    deps = [":HelloLib"],
-)
-
-# From: `examples/testing/multi_frameworks_toolchain/example/BUILD`
-scala_test(
-    name = "scalatest_example",
-    srcs = ["ScalaTestExampleTest.scala"],
-)
+load(":defs.bzl", "scalafmt_scala_test")
 
-scala_specs2_junit_test(
-    name = "specs2_example",
-    srcs = ["Specs2ExampleTest.scala"],
-    suffixes = ["Test"],
-)
-
-# Manufactured based on `docs/phase_scalafmt.md` and `test/scalafmt/BUILD`.
+# Based on `docs/phase_scalafmt.md`, `test/scalafmt/BUILD`, and a copy of:
+# examples/testing/multi_frameworks_toolchain/example/ScalaTestExampleTest.scala
 scalafmt_scala_test(
     name = "ScalafmtTest",
     srcs = ["ScalaTestExampleTest.scala"],
     format = True,
 )
-
-# From: `test/proto/BUILD`
-proto_library(
-    name = "standalone_proto",
-    srcs = ["standalone.proto"],
-)
-
-scala_proto_library(
-    name = "standalone_scala_proto",
-    deps = [":standalone_proto"],
-)
-
-default_outputs_test(
-    name = "standalone_scala_proto_outs_test",
-    expected_outs = [
-        "standalone_proto_scalapb-src.jar",
-        "standalone_proto_scalapb.jar",
-    ],
-    target_under_test = ":standalone_scala_proto",
-)
-
-# From: `test/jmh/BUILD`
-java_library(
-    name = "java_type",
-    srcs = ["JavaType.java"],
-    visibility = ["//visibility:public"],
-)
-
-scala_library(
-    name = "scala_type",
-    srcs = ["ScalaType.scala"],
-    visibility = ["//visibility:public"],
-)
-
-scala_library(
-    name = "add_numbers",
-    srcs = ["AddNumbers.scala"],
-    visibility = ["//visibility:public"],
-    exports = [
-        ":java_type",
-        ":scala_type",
-    ],
-    deps = [
-        ":java_type",
-        ":scala_type",
-    ],
-)
-
-scala_benchmark_jmh(
-    name = "test_benchmark",
-    srcs = ["TestBenchmark.scala"],
-    data = ["data.txt"],
-    deps = [":add_numbers"],
-)
-
-# From: `test/src/main/scala/scalarules/test/twitter_scrooge/BUILD`
-
-thrift_library(
-    name = "thrift3",
-    srcs = ["Thrift3.thrift"],
-    visibility = ["//visibility:public"],
-)
-
-scrooge_scala_library(
-    name = "scrooge3",
-    visibility = ["//visibility:public"],
-    deps = [":thrift3"],
-)
-
-scrooge_java_library(
-    name = "scrooge3_java",
-    visibility = ["//visibility:public"],
-    deps = [":thrift3"],
-)
-
-scrooge_transitive_outputs_test(
-    name = "scrooge_test_scala",
-    dep = ":scrooge3",
-    expected_jars = ["thrift3_scrooge_scala.jar"],
-)
-
-scrooge_transitive_outputs_test(
-    name = "scrooge_test_java",
-    dep = ":scrooge3_java",
-    expected_jars = ["thrift3_scrooge_java.jar"],
-)

deps/test/HelloLib.scala

@@ -1,16 +1,19 @@
 """Bazel module template for test/shell/test_deps_versions.sh tests."""
 
-module(
-    name = "rules_scala_deps_versions_test",
-    bazel_compatibility = [">=${bazelversion}"],
-)
+module(name = "rules_scala_deps_versions_test")
 
 bazel_dep(name = "rules_scala")
 local_path_override(
     module_name = "rules_scala",
     path = "../..",
 )
 
+bazel_dep(name = "multi_frameworks_toolchain")
+local_path_override(
+    module_name = "multi_frameworks_toolchain",
+    path = "../../examples/testing/multi_frameworks_toolchain",
+)
+
 bazel_dep(name = "bazel_skylib")
 single_version_override(
     module_name = "bazel_skylib",

deps/test/MODULE.bazel.template

@@ -1,47 +1,7 @@
-load("@bazel_skylib//lib:collections.bzl", "collections")
-load("@bazel_skylib//lib:unittest.bzl", "analysistest", "asserts", "unittest")
-load("@rules_java//java/common:java_info.bzl", "JavaInfo")
+"""Utilities required for dependency compatibility tests."""
+
 load("@rules_scala//scala:advanced_usage/scala.bzl", "make_scala_test")
 load("@rules_scala//scala/scalafmt:phase_scalafmt_ext.bzl", "ext_scalafmt")
 
 # From //test/scalafmt:phase_scalafmt_test.bzl
 scalafmt_scala_test = make_scala_test(ext_scalafmt)
-
-# From //test/proto:default_outputs_test.bzl
-def _default_outputs_test(ctx):
-    env = analysistest.begin(ctx)
-
-    target_under_test = analysistest.target_under_test(env)
-    actual_outs = [f.basename for f in target_under_test[DefaultInfo].files.to_list()]
-
-    asserts.equals(env, sorted(ctx.attr.expected_outs), sorted(actual_outs))
-
-    return analysistest.end(env)
-
-default_outputs_test = analysistest.make(
-    _default_outputs_test,
-    attrs = {
-        "expected_outs": attr.string_list(),
-    },
-)
-
-# From
-# //test/src/main/scala/scalarules/test/twitter_scrooge:twitter_scrooge_test.bzl
-def _scrooge_transitive_outputs(ctx):
-    env = unittest.begin(ctx)
-
-    asserts.equals(
-        env,
-        sorted(ctx.attr.expected_jars),
-        sorted(collections.uniq([out.class_jar.basename for out in ctx.attr.dep[JavaInfo].outputs.jars])),
-    )
-
-    return unittest.end(env)
-
-scrooge_transitive_outputs_test = unittest.make(
-    _scrooge_transitive_outputs,
-    attrs = {
-        "dep": attr.label(),
-        "expected_jars": attr.string_list(),
-    },
-)

deps/test/ScalaBinary.scala

@@ -8,7 +8,7 @@ local_path_override(
     path = "../..",
 )
 
-bazel_dep(name = "latest_dependencies")
+bazel_dep(name = "latest_dependencies", dev_dependency = True)
 local_path_override(
     module_name = "latest_dependencies",
     path = "../../deps/latest",

deps/test/defs.bzl

@@ -8,7 +8,7 @@ local_path_override(
     path = "../..",
 )
 
-bazel_dep(name = "latest_dependencies")
+bazel_dep(name = "latest_dependencies", dev_dependency = True)
 local_path_override(
     module_name = "latest_dependencies",
     path = "../../deps/latest",

dt_patches/test_dt_patches/MODULE.bazel

@@ -8,7 +8,7 @@ local_path_override(
     path = "../..",
 )
 
-bazel_dep(name = "latest_dependencies")
+bazel_dep(name = "latest_dependencies", dev_dependency = True)
 local_path_override(
     module_name = "latest_dependencies",
     path = "../../deps/latest",

dt_patches/test_dt_patches_user_srcjar/MODULE.bazel

@@ -8,7 +8,7 @@ local_path_override(
     path = "../..",
 )
 
-bazel_dep(name = "latest_dependencies")
+bazel_dep(name = "latest_dependencies", dev_dependency = True)
 local_path_override(
     module_name = "latest_dependencies",
     path = "../../deps/latest",

examples/crossbuild/MODULE.bazel

@@ -8,7 +8,7 @@ local_path_override(
     path = "../..",
 )
 
-bazel_dep(name = "latest_dependencies")
+bazel_dep(name = "latest_dependencies", dev_dependency = True)
 local_path_override(
     module_name = "latest_dependencies",
     path = "../../deps/latest",