Fix last_green w/bazel_worker_api, rules_java 8.14

mbland · mbland · commit 4daa60819356 · 2025-07-26T16:37:07.000-04:00
Uses the `bazel_worker_{api,java}` modules and a legacy `WORKSPACE` shim to replace `@bazel_tools//src/main/protobuf:worker_protocol_java_proto`. This, and bumping `rules_java` to 8.14.0, fixes breakages in the latest `last_green` Bazel at commit 00b48b27aa216befc419e540919a1f1df4cda337. Also: - Removes the `--incompatible_autoload_externally` flag from `.bazelrc`. - Fixes `_get_compiler_srcjar` to return empty `compiler_srcjar_objects` instead of returning a valid object. `test_compiler_srcjar_error 2.12.11` from `dt_patches/dt_patch_test.sh` caught this under `WORKSPACE`. Added `test_fail_if_compiler_srcjar_object_is_empty` to `test/shell/test_compiler_sources_integrity.sh` to catch the specific problem sooner. - Fixed regex in `target_file_location` from `test_helper.sh` to work with Bazel 6.5.0, and fixed unbound variable in `fail` message. - Added logic to `teardown_suite` in `test_bzlmod_macros.sh` to avoid unbound variable breakage when running under Bazel 6.5.0. - Added logic to skip `test_check_module_bazel_template` under Bazel 6 in `test_version.sh`. - Removed restriction to use `protobuf` v29 or lower with Bazel 6 from `README.md`. - Added `scala_compiler_source.*` to the regular expressions for "canonical reproducible" messages in `dt_patches/dt_patch_test.sh`. This resolved a test breakage when `rules_python` began to emit a "canonical reproducible" message. (Also changed `--expunge` to `--expunge_async`.) --- bazelbuild/bazel#26477, which also landed in the 9.0.0-pre.20250714.1 rolling release, broke the `last_green` build: ``` $ USE_BAZEL_VERSION=9.0.0-pre.20250714.1 bazel test //src/... //test/... ERROR: error loading package '@@bazel_tools//src/main/protobuf': Unable to find package for @@[unknown repo 'grpc-java' requested from @@bazel_tools]//:java_grpc_library.bzl: The repository '@@[unknown repo 'grpc-java' requested from @@bazel_tools]' could not be resolved: No repository visible as '@grpc-java' from repository '@@bazel_tools'. ``` The failing target, and several others, depended upon `@bazel_tools//src/main/protobuf:worker_protocol_java_proto`. Nothing else depends on any other target from that package. So upon @Wyverald's advice, I attempted to use the `bazel_worker_java` module to replace this specific dependency: - bazelbuild/bazel#26579 (comment) Using just `bazel_worker_java` 0.0.5 at first produced this breakage: ```txt $ USE_BAZEL_VERSION=last_green bazel build //src/... //test/... ERROR: .../external/bazel_worker_api+/BUILD.bazel:3:1: name 'proto_library' is not defined ``` This is because that version of `bazel_worker_java` still depends upon `bazel_worker_api` 0.0.1, which didn't load `proto_library` from `@com_google_protobuf`. Adding `bazel_worker_api` 0.0.5 directly to `MODULE.bazel` fixed this. The next failure was due to `rules_scala` overriding `rules_java` to be 8.12.0, whereas `last_green` now requires 8.14.0: ```txt $ USE_BAZEL_VERSION=last_green bazel build //src/... //test/... ERROR: src/java/io/bazel/rulesscala/jar/BUILD:3:13: in java_library rule //src/java/io/bazel/rulesscala/jar:jar: Traceback (most recent call last): File ".../external/rules_java+/java/bazel/rules/bazel_java_library.bzl", line 26, column 35, in _proxy return bazel_java_library_rule( File ".../external/rules_java+/java/common/rules/impl/bazel_java_library_impl.bzl", line 68, column 43, in bazel_java_library_rule target, base_info = basic_java_library( File ".../external/rules_java+/java/common/rules/impl/basic_java_library_impl.bzl", line 127, column 49, in basic_java_library java_info, compilation_info = compile_action( File ".../external/rules_java+/java/common/rules/impl/compile_action.bzl", line 139, column 46, in compile_action java_info = _compile_private_for_builtins( File ".../external/rules_java+/java/private/java_common_internal.bzl", line 217, column 68, in compile get_internal_java_common().create_header_compilation_action( Error in create_header_compilation_action: create_header_compilation_action() missing 2 required positional arguments: header_compilation_jar, header_compilation_direct_deps ERROR: src/java/io/bazel/rulesscala/jar/BUILD:3:13: Analysis of target '//src/java/io/bazel/rulesscala/jar:jar' failed ``` Bumping `rules_java` to 8.14.0 in the `single_version_override` fixed that. Finally, since there's no legacy `WORKSPACE` API in bazelbuild/bazel-worker-api, I had to roll my own new `workspace_compat()` macro. This macro creates `@bazel_worker_{api,java}` repos with aliases to targets in `@bazel-worker-api`. Regarding `--incompatible_autoload_externally`, I'd added it in bazel-contrib#1748 when fixing problems with `WORKSPACE` builds under Bazel 8.2.1. However, it's not really necessary, so I removed it.
diff --git a/.bazelrc b/.bazelrc
@@ -1,5 +1,5 @@
 # Remove once Bazel 8 becomes the minimum supported version.
-common --noenable_workspace --incompatible_use_plus_in_repo_names --incompatible_autoload_externally=
+common --noenable_workspace --incompatible_use_plus_in_repo_names
 
 # Uncomment to run tests under `WORKSPACE`. Remove once Bazel 9 becomes the
 # minimum supported version.
diff --git a/MODULE.bazel b/MODULE.bazel
@@ -42,7 +42,7 @@ single_version_override(
 bazel_dep(name = "rules_java", version = "7.6.0")
 single_version_override(
     module_name = "rules_java",
-    version = "8.12.0",
+    version = "8.14.0",
 )
 
 bazel_dep(name = "rules_proto", version = "6.0.0")
@@ -66,6 +66,9 @@ single_version_override(
     version = "31.1",
 )
 
+bazel_dep(name = "bazel_worker_api", version = "0.0.5")
+bazel_dep(name = "bazel_worker_java", version = "0.0.5")
+
 scala_protoc = use_extension(
     "//scala/extensions:protoc.bzl",
     "scala_protoc",
diff --git a/README.md b/README.md
@@ -1318,36 +1318,12 @@ future compatibility.
 
 __`rules_scala` 7.x officially drops support for Bazel 6.5.0.__ Bzlmod builds
 with Bazel 6.5.0 won't work at all because [Bazel 6.5.0 doesn't support
-'use_repo_rule']( https://bazel.build/versions/6.5.0/rules/lib/globals), which
+'use_repo_rule'](https://bazel.build/versions/6.5.0/rules/lib/globals), which
 ['rules_jvm_external' >= 6.3 requires](
 https://github.com/bazelbuild/rules_scala/issues/1482#issuecomment-2515496234).
 
 At the moment, `WORKSPACE` builds mostly continue to work with Bazel 6.5.0, but
-not out of the box, and may break at any time.
-
-#### Maximum of `protobuf` v29
-
-You _must_ use `protobuf` v29 or earlier. `rules_scala` now uses v30 by default,
-which removes `py_proto_library` and other symbols that Bazel 6.5.0 requires:
-
-```txt
-ERROR: Traceback (most recent call last):
-  File ".../external/bazel_tools/src/main/protobuf/BUILD",
-  line 1, column 46, in <toplevel>
-    load("@com_google_protobuf//:protobuf.bzl", "py_proto_library")
-
-Error: file '@com_google_protobuf//:protobuf.bzl'
-  does not contain symbol 'py_proto_library'
-
-ERROR: .../src/java/io/bazel/rulesscala/worker/BUILD:3:13:
-  no such target '@bazel_tools//src/main/protobuf:worker_protocol_java_proto':
-  target 'worker_protocol_java_proto'
-  not declared in package 'src/main/protobuf'
-  defined by .../external/bazel_tools/src/main/protobuf/BUILD
-  (Tip: use `query "@bazel_tools//src/main/protobuf:*"`
-    to see all the targets in that package)
-  and referenced by '//src/java/io/bazel/rulesscala/worker:worker'
-```
+may break at any time.
 
 #### Configuring the protocol compiler toolchain
 
diff --git a/deps/latest/MODULE.bazel b/deps/latest/MODULE.bazel
@@ -9,7 +9,7 @@ module(
 bazel_dep(name = "bazel_skylib", version = "1.7.1")
 bazel_dep(name = "platforms", version = "1.0.0")
 bazel_dep(name = "protobuf", version = "31.1")
-bazel_dep(name = "rules_java", version = "8.12.0")
+bazel_dep(name = "rules_java", version = "8.14.0")
 bazel_dep(name = "rules_proto", version = "7.1.0")
 
 # https://github.com/bazelbuild/bazel/pull/25681 removed
diff --git a/dt_patches/dt_patch_test.sh b/dt_patches/dt_patch_test.sh
@@ -36,7 +36,7 @@ test_compiler_srcjar() {
   run_in_test_repo \
     test_dt_patches_user_srcjar \
     bazel build "--repo_env=SCALA_VERSION=${SCALA_VERSION}" //... 2>&1 |
-    (! grep "canonical reproducible")
+    (! grep "scala_compiler_source.*canonical reproducible")
 }
 
 test_compiler_srcjar_nonhermetic() {
@@ -46,7 +46,7 @@ test_compiler_srcjar_nonhermetic() {
   run_in_test_repo \
     test_dt_patches_user_srcjar \
     bazel build "--repo_env=SCALA_VERSION=${SCALA_VERSION}" //... 2>&1 |
-    grep 'canonical reproducible'
+    grep 'scala_compiler_source.*canonical reproducible'
 }
 
 test_compiler_srcjar_error() {
@@ -129,7 +129,7 @@ $runner test_compiler_srcjar_error 2.12.13
 # that we grep for will only be outputted the first time (on Bazel >= 6).
 # So we clean the repo first to ensure consistency.
 
-run_in_test_repo 'test_dt_patches_user_srcjar' bazel clean --expunge
+run_in_test_repo 'test_dt_patches_user_srcjar' bazel clean --expunge_async
 
 $runner test_compiler_srcjar 2.12.14
 $runner test_compiler_srcjar 2.12.15
diff --git a/scala/deps.bzl b/scala/deps.bzl
@@ -2,6 +2,7 @@
 
 load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
 load("@bazel_tools//tools/build_defs/repo:utils.bzl", "maybe")
+load("//scala/private:macros/workspace_compat.bzl", "workspace_compat")
 
 def rules_scala_dependencies():
     """Instantiates repos needed by rules provided by `rules_scala`."""
@@ -64,3 +65,5 @@ def rules_scala_dependencies():
         strip_prefix = "rules_proto-6.0.2",
         url = "https://github.com/bazelbuild/rules_proto/releases/download/6.0.2/rules_proto-6.0.2.tar.gz",
     )
+
+    workspace_compat()
diff --git a/scala/latest_deps.bzl b/scala/latest_deps.bzl
@@ -2,6 +2,7 @@
 
 load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
 load("@bazel_tools//tools/build_defs/repo:utils.bzl", "maybe")
+load("//scala/private:macros/workspace_compat.bzl", "workspace_compat")
 
 def rules_scala_dependencies():
     """Instantiates repos needed by rules provided by `rules_scala`."""
@@ -29,9 +30,9 @@ def rules_scala_dependencies():
         http_archive,
         name = "rules_java",
         urls = [
-            "https://github.com/bazelbuild/rules_java/releases/download/8.12.0/rules_java-8.12.0.tar.gz",
+            "https://github.com/bazelbuild/rules_java/releases/download/8.14.0/rules_java-8.14.0.tar.gz",
         ],
-        sha256 = "1558508fc6c348d7f99477bd21681e5746936f15f0436b5f4233e30832a590f9",
+        sha256 = "bbe7d94360cc9ed4607ec5fd94995fd1ec41e84257020b6f09e64055281ecb12",
     )
 
     maybe(
@@ -69,3 +70,5 @@ def rules_scala_dependencies():
             rules_jvm_external_tag,
         ),
     )
+
+    workspace_compat()
diff --git a/scala/private/macros/scala_repositories.bzl b/scala/private/macros/scala_repositories.bzl
@@ -58,7 +58,7 @@ compiler_sources_repo = repository_rule(
 )
 
 def _get_compiler_srcjar(scala_version, scala_compiler_srcjar):
-    if scala_compiler_srcjar:
+    if scala_compiler_srcjar != None:
         return scala_compiler_srcjar
 
     compiler_srcjar = COMPILER_SOURCES.get(scala_version, None)
diff --git a/scala/private/macros/workspace_compat.bzl b/scala/private/macros/workspace_compat.bzl
@@ -0,0 +1,64 @@
+"""Macro to translate Bazel modules into legacy WORKSPACE compatible repos
+
+Used only for Bazel modules that don't offer a legacy WORKSPACE compatible API
+already. Originally became necessary due to:
+
+- https://github.com/bazelbuild/bazel/issues/26579#issuecomment-3120862995
+"""
+
+load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
+load("@bazel_tools//tools/build_defs/repo:utils.bzl", "maybe")
+
+_ALIAS_REPO_BUILD_FMT = """\
+package(default_visibility = ["//visibility:public"])
+
+{aliases}"""
+
+_ALIAS_FMT = """\
+alias(
+    name = "{name}",
+    actual = "{actual}",
+)
+"""
+
+def _alias_repo_impl(repository_ctx):
+    aliases = "\n".join([
+        _ALIAS_FMT.format(name = name, actual = actual)
+        for name, actual in repository_ctx.attr.aliases.items()
+    ])
+    repository_ctx.file(
+        "BUILD",
+        _ALIAS_REPO_BUILD_FMT.format(aliases = aliases),
+        executable = False,
+    )
+
+_alias_repo = repository_rule(
+    implementation = _alias_repo_impl,
+    attrs = {"aliases": attr.string_dict(mandatory = True)},
+)
+
+def workspace_compat():
+    maybe(
+        http_archive,
+        name = "bazel-worker-api",
+        sha256 = "07b5fa9c883558e378a00c67dae05a4dcb8d43cc12486722d6875224454a8a30",
+        urls = [
+            "https://mirror.bazel.build/github.com/bazelbuild/bazel-worker-api/releases/download/v0.0.5/bazel-worker-api-v0.0.5.tar.gz",
+            "https://github.com/bazelbuild/bazel-worker-api/releases/download/v0.0.5/bazel-worker-api-v0.0.5.tar.gz",
+        ],
+        strip_prefix = "bazel-worker-api-0.0.5",
+    )
+
+    _alias_repo(
+        name = "bazel_worker_api",
+        aliases = {
+            "worker_protocol_proto": "@bazel-worker-api//proto:worker_protocol_proto",
+        },
+    )
+
+    _alias_repo(
+        name = "bazel_worker_java",
+        aliases = {
+            "worker_protocol_java_proto": "@bazel-worker-api//java:worker_protocol_java_proto",
+        },
+    )
diff --git a/src/java/io/bazel/rulesscala/coverage/instrumenter/BUILD b/src/java/io/bazel/rulesscala/coverage/instrumenter/BUILD
@@ -15,8 +15,8 @@ java_binary(
         "//src/java/io/bazel/rulesscala/io_utils",
         "//src/java/io/bazel/rulesscala/jar",
         "//src/java/io/bazel/rulesscala/worker",
-        "@bazel_tools//src/main/protobuf:worker_protocol_java_proto",
         "@bazel_tools//tools/jdk:JacocoCoverage",
+        "@bazel_worker_java//:worker_protocol_java_proto",
     ],
 )
 
diff --git a/src/java/io/bazel/rulesscala/scalac/definitions.bzl b/src/java/io/bazel/rulesscala/scalac/definitions.bzl
@@ -11,7 +11,7 @@ DEFAULT_SCALAC_DEPS = [
         "//src/protobuf/io/bazel/rules_scala:diagnostics_java_proto",
         "//src/java/io/bazel/rulesscala/scalac/compileoptions",
         "//src/java/io/bazel/rulesscala/scalac/reporter",
-        "@bazel_tools//src/main/protobuf:worker_protocol_java_proto",
+        "@bazel_worker_java//:worker_protocol_java_proto",
     ]
 ]
 
diff --git a/src/java/io/bazel/rulesscala/worker/BUILD b/src/java/io/bazel/rulesscala/worker/BUILD
@@ -5,7 +5,7 @@ java_library(
     srcs = ["Worker.java"],
     visibility = ["//visibility:public"],
     deps = [
-        "@bazel_tools//src/main/protobuf:worker_protocol_java_proto",
+        "@bazel_worker_java//:worker_protocol_java_proto",
     ],
 )
 
@@ -20,6 +20,6 @@ java_test(
     test_class = "io.bazel.rulesscala.worker.WorkerTest",
     deps = [
         ":worker",
-        "@bazel_tools//src/main/protobuf:worker_protocol_java_proto",
+        "@bazel_worker_java//:worker_protocol_java_proto",
     ],
 )
diff --git a/test/BUILD b/test/BUILD
@@ -549,8 +549,8 @@ scala_specs2_junit_test(
     name = "data_location_expansion",
     size = "small",
     srcs = ["src/main/scala/scalarules/test/location_expansion/LocationExpansionTest.scala"],
-    data = ["@bazel_tools//src/main/protobuf:worker_protocol_java_proto"],
-    jvm_flags = ["-Dlocation.expanded='$(locations @bazel_tools//src/main/protobuf:worker_protocol_java_proto)'"],
+    data = ["@bazel_worker_java//:worker_protocol_java_proto"],
+    jvm_flags = ["-Dlocation.expanded='$(locations @bazel_worker_java//:worker_protocol_java_proto)'"],
     suffixes = ["Test"],
 )
 
diff --git a/test/compiler_sources_integrity/MODULE.bazel b/test/compiler_sources_integrity/MODULE.bazel
@@ -39,6 +39,11 @@ scala_deps.compiler_srcjar(
     ),
     version = "3.1.999",
 )
+
+# Used for test_fail_if_compiler_srcjar_object_is_empty.
+scala_deps.compiler_srcjar(
+    version = "3.1.1000",
+)
 use_repo(
     scala_deps,
     "scala_compiler_sources",
diff --git a/test/compiler_sources_integrity/WORKSPACE b/test/compiler_sources_integrity/WORKSPACE
@@ -49,6 +49,7 @@ scala_toolchains(
                 "scala3-compiler_3/3.1.3/scala3-compiler_3-3.1.3-sources.jar"
             ),
         },
+        "3.1.1000": {},
     },
     # Since we're using a bogus Scala version in the compiler_srcjar.
     validate_scala_version = False,
diff --git a/test/shell/test_bzlmod_macros.sh b/test/shell/test_bzlmod_macros.sh
@@ -40,7 +40,9 @@ setup_suite() {
 }
 
 teardown_suite() {
-    teardown_test_tmpdir "$original_dir" "$test_tmpdir"
+    if [[ -n "${original_dir:-}" ]];  then
+      teardown_test_tmpdir "$original_dir" "$test_tmpdir"
+    fi
 }
 
 setup_test_module() {
diff --git a/test/shell/test_compiler_sources_integrity.sh b/test/shell/test_compiler_sources_integrity.sh
@@ -188,6 +188,21 @@ test_fail_with_scala3_compiler_source_link_for_unknown_major_version() {
   _expect_failure_with_guessed_url "$scala_version" "$guessed_url"
 }
 
+test_fail_if_compiler_srcjar_object_is_empty() {
+  # This case is covered more in depth in dt_patches/dt_patch_test.sh via its
+  # test_compiler_srcjar_error helper function. However, we did have a bug
+  # whereby _get_compiler_srcjar incorrectly returned a valid value when its
+  # scala_compiler_srcjar argument was empty. Therefore it's worth testing here.
+  local expected_msg=(
+    "scala_compiler_srcjar invalid, must be a dict with exactly one of"
+    '"label", "url" or "urls" keys, got: '
+  )
+
+  action_should_fail_with_message \
+    "${expected_msg[*]}" \
+    build --repo_env=SCALA_VERSION=3.1.1000 //...
+}
+
 # main()
 
 setup_suite
diff --git a/test/shell/test_helper.sh b/test/shell/test_helper.sh
@@ -194,15 +194,15 @@ fail() {
 target_file_location() {
   local target="$1"
   local cmd=(bazel query --output location "$target")
-  local regex="([^[:cntrl:]]*):[0-9]+:[0-9]+: [^[:cntrl:]]+ ${target//./\\.}$"
+  local regex="([^[:cntrl:]]*):[0-9]+:[0-9]+: [^[:cntrl:]]+ ${target//./\\.}"
   local output
 
   if ! output="$("${cmd[@]}" 2>&1)"; then
     fail "failed: ${cmd[*]}" "$output"
   elif [[ "$output" =~ $regex ]]; then
     printf '%s' "${BASH_REMATCH[1]}"
   else
-    fail "output of: ${cmd[*]}" "does not match: ${pattern}" "$output"
+    fail "output of: ${cmd[*]}" "does not match: ${regex}" "$output"
   fi
 }
 
diff --git a/test_version.sh b/test_version.sh
@@ -98,6 +98,12 @@ check_module_bazel_template() {
 }
 
 test_check_module_bazel_template() {
+  if [[ "$(bazel --version)" =~ ^bazel\ 6\. ]]; then
+    local msg="${FUNCNAME} not compatible with Bazel 6, skipping..."
+    echo -e " ${YELLOW}${msg}${NC}"
+    return
+  fi
+
   run_in_test_repo "check_module_bazel_template" \
     "bzlmod_tidy" \
     "version_specific_tests_dir/"
diff --git a/test_version/MODULE.bazel.template b/test_version/MODULE.bazel.template
@@ -14,9 +14,9 @@ local_path_override(
     path = "../../deps/latest",
 )
 
-bazel_dep(name = "rules_java", version = "8.11.0")
+bazel_dep(name = "rules_java", version = "8.14.0")
 bazel_dep(name = "rules_proto", version = "7.1.0")
-bazel_dep(name = "rules_shell", version = "0.4.1")
+bazel_dep(name = "rules_shell", version = "0.5.0")
 bazel_dep(
     name = "protobuf",
     version = "31.1",

Original file line number	Diff line number	Diff line change
`@@ -58,7 +58,7 @@ compiler_sources_repo = repository_rule(`
`58`	`58`	`)`
`59`	`59`
`60`	`60`	`def _get_compiler_srcjar(scala_version, scala_compiler_srcjar):`
`61`		`- if scala_compiler_srcjar:`
	`61`	`+ if scala_compiler_srcjar != None:`
`62`	`62`	`return scala_compiler_srcjar`
`63`	`63`
`64`	`64`	`compiler_srcjar = COMPILER_SOURCES.get(scala_version, None)`
Original file line number	Diff line number	Diff line change
`@@ -15,8 +15,8 @@ java_binary(`
`15`	`15`	`"//src/java/io/bazel/rulesscala/io_utils",`
`16`	`16`	`"//src/java/io/bazel/rulesscala/jar",`
`17`	`17`	`"//src/java/io/bazel/rulesscala/worker",`
`18`		`- "@bazel_tools//src/main/protobuf:worker_protocol_java_proto",`
`19`	`18`	`"@bazel_tools//tools/jdk:JacocoCoverage",`
	`19`	`+ "@bazel_worker_java//:worker_protocol_java_proto",`
`20`	`20`	`],`
`21`	`21`	`)`
`22`	`22`