Update to use Starlark rules_android. Reapply ba7310ce4a1d8fb14434597fbc7440a4074f7695

[ahumesky]

Reverts #1215 plus accounting for additional changes since ba7310c went in

[jin]

Thanks for pushing this through! The failing tests look legitimate.

[ahumesky]

Thanks for taking a look, this draft is to run the CI to see what fails

[ahumesky]

I have the failing tests down to:

• //tests/unit/kotlin:inline_function_test on bazel 8.0.0 (https://buildkite.com/bazel/rules-jvm-external/builds/4759#019448d2-531d-4594-8dbf-b97d3ab6cfca) I can't readily reproduce this error, it passes for me locally, so I'll have to see what's different between my setup and the test setup.
• //... on bazel 6.4.0 (https://buildkite.com/bazel/rules-jvm-external/builds/4759#019448d2-531b-4335-a606-65be93992deb). I can reproduce this locally and is somewhat expected because I haven't looked closely at making the tests with bazel 6.x use the native rules.

[ahumesky]

The //tests/unit/kotlin:inline_function_test failure with bazel 8.0.0 is due to disabling --experimental_sibling_repository_layout, and of course restoring that causes the original failures with the go rules (bazel-contrib/rules_go#3947)

[jin]

There is not enough motivation to flip --experimental_sibling_repository_layout to true so we should be ok to remove it from rules_jvm_external's bazelrc (bazelbuild/bazel#20500 (comment))

And bumping rules_kotlin version fixes the test failure that comes with removing the sibling repository layout flag:

INFO: Analyzed target //tests/unit/kotlin:inline_function_test (114 packages loaded, 6285 targets configured).
INFO: Found 1 test target...
Target //tests/unit/kotlin:inline_function_test up-to-date:
  bazel-bin/tests/unit/kotlin/inline_function_test.jar
  bazel-bin/tests/unit/kotlin/inline_function_test.jdeps
INFO: Elapsed time: 43.089s, Critical Path: 2.71s
INFO: 12 processes: 40 action cache hit, 8 internal, 2 darwin-sandbox, 2 worker.
INFO: Build completed successfully, 12 total actions
//tests/unit/kotlin:inline_function_test                                 PASSED in 0.3s

Executed 1 out of 1 test: 1 test passes.
There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.


jingwen@jingwen-mac rules_jvm_external % git diff
diff --git a/.bazelrc b/.bazelrc
index d137de4..6f82835 100644
--- a/.bazelrc
+++ b/.bazelrc
@@ -8,7 +8,7 @@ build --experimental_strict_java_deps=strict
 build --explicit_java_test_deps

 # Re-enable once https://github.com/bazelbuild/rules_go/issues/3947 is addressed.
-build --experimental_sibling_repository_layout
+# build --experimental_sibling_repository_layout

 # Make sure we get something helpful when tests fail
 test --verbose_failures
diff --git a/MODULE.bazel b/MODULE.bazel
index 60a0dcd..1c6a015 100644
--- a/MODULE.bazel
+++ b/MODULE.bazel
@@ -30,7 +30,7 @@ bazel_dep(
 )
 bazel_dep(
     name = "rules_kotlin",
-    version = "1.9.6",
+    version = "2.1.0",
 )
 bazel_dep(
     name = "rules_shell",

[ahumesky]

Awesome thanks for taking a look at this -- I was initially looking at rules_java since the error was about the path to the java binary (/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/ec321eb2cc2d0f8f91b676b6d4c66c29/sandbox/linux-sandbox/1594/execroot/_main/bazel-out/k8-fastbuild/bin/tests/unit/kotlin/inline_function_test.runfiles/_main/external/rules_java++toolchains+remotejdk11_linux/bin/java: No such file or directory)

Unfortunately updating rules_kotlin to 2.1.0 seems to cause a different (yet still path related) failure on windows:

set USE_BAZEL_VERSION=8.0.0
bazelisk build //tests/integration/kt_jvm_export:test-lib

LAUNCHER ERROR: Rlocation failed on _main/external/rules_kotlin++rules_kotlin_extensions+com_github_jetbrains_kotlin_git/lib/annotations-13.0.jar:external/rules_kotlin++rules_kotlin_extensions+com_github_jetbrains_kotlin_git/lib/kotlin-stdlib.jar:external/rules_kotlin++rules_kotlin_extensions+com_github_jetbrains_kotlin_git/lib/kotlin-stdlib-jdk7.jar:external/rules_kotlin++rules_kotlin_extensions+com_github_jetbrains_kotlin_git/lib/kotlin-stdlib-jdk8.jar, path doesn't exist in MANIFEST file

Switching to rules_kotlin 2.0.0 seems to avoid that error, so it must be one of the changes between 2.0.0 and 2.1.0: bazelbuild/rules_kotlin@v2.0.0...v2.1.0

[ahumesky]

Ah, but 2.0.0 still has the No such file or directory problem with bin/java on linux: https://buildkite.com/bazel/rules-jvm-external/builds/4767#01946284-2045-4179-af17-cec4ce769182. Bummer. I'll try to find which commit between 2.0.0 and 2.1.0 breaks things on windows tomorrow

[ahumesky]

The issue with rules_kotlin appears to be from this change: bazelbuild/rules_kotlin@e51619c

Although the error is coming from code that was introduced earlier than that commit:
https://github.com/bazelbuild/rules_kotlin/blob/f20ef32db6988fa1b346b1af12f3a3edd545bc47/src/main/starlark/core/compile/cli/compile.bzl#L33-L35

Maybe that code was not being used until bazelbuild/rules_kotlin@e51619c, not sure

It seems that the java launcher separates these by semicolon:
https://github.com/bazelbuild/bazel/blob/bebe20ed5078edf123b7d528931658168df2cb4c/src/tools/launcher/java_launcher.cc#L328-L331

However changing the Starlark code to use semicolon still results in

LAUNCHER ERROR: Rlocation failed on _main/external/rules_kotlin++rules_kotlin_extensions+com_github_jetbrains_kotlin_git/lib/annotations-13.0.jar, path doesn't exist in MANIFEST file

And the code does appear to add these files to the inputs:
https://github.com/bazelbuild/rules_kotlin/blob/f20ef32db6988fa1b346b1af12f3a3edd545bc47/src/main/starlark/core/compile/cli/compile.bzl#L28

[ahumesky]

I also noticed that rules_kotlin added these recently to its .bazelrc:

# Required for windows ci
startup --windows_enable_symlinks
common --enable_runfiles

https://github.com/bazelbuild/rules_kotlin/blob/f20ef32db6988fa1b346b1af12f3a3edd545bc47/.bazelrc#L12-L14

but that didn't seem to help the problem

[jin]

@ahumesky what's the status of this and active blockers to move off rules_android 0.1.1?

[ahumesky]

Thanks I should be able to pick this back up this week.

[ahumesky]

I have some fixes to rules_kotlin that gets the Windows tests in rules_jvm_external to pass: bazelbuild/rules_kotlin#1274

Unfortunately as-is it breaks a bunch of other stuff so I'll have to see how to fix that fix...

This also means we'll need to wait for a rules_kotlin release that includes these fixes.

It's not clear to me offhand how supported Windows is in rules_kotlin, the tests running on Windows are disabled: https://github.com/bazelbuild/rules_kotlin/blob/096170ffd892817044d56e8959c5e73a5856a98d/.bazelci/presubmit.yml#L10

[ahumesky]

Temporarily brought my fork of rules_kotlin into this PR (c97dd30) to test bazelbuild/rules_kotlin#1274 and it appears to be working.

The last thing to fix in this PR is to not try to run the Starlark Android rules with bazel 6.4.0

Then we'll have to wait for a rules_kotlin release with bazelbuild/rules_kotlin#1274

[ahumesky]

All the tests are passing now (including those on bazel 6.4.0 which will use the native android rules) with my update to rules_kotlin (bazelbuild/rules_kotlin#1274). I'll leave this as a draft until there's a release of rules_kotlin with my PR.

[ahumesky]

Pulling in the latest changes, 0b5d9da updates rules_jvm_external to protobuf 29.3, which seems to conflict with the version rules_android uses: https://github.com/bazelbuild/rules_android/blob/de5a97fa897ce244fec8468a9eb411f8d88a8e0e/defs.bzl#L70-L71

resulting in

Exception in thread "main" java.lang.ExceptionInInitializerError
	at com.google.devtools.build.android.AndroidDataSerializer.flushTo(AndroidDataSerializer.java:71)
	at com.google.devtools.build.android.AndroidResourceParsingAction.main(AndroidResourceParsingAction.java:80)
	at com.google.devtools.build.android.ResourceProcessorBusyBox$Tool$2.call(ResourceProcessorBusyBox.java:74)
	at com.google.devtools.build.android.ResourceProcessorBusyBox.processRequest(ResourceProcessorBusyBox.java:238)
	at com.google.devtools.build.android.ResourceProcessorBusyBox.main(ResourceProcessorBusyBox.java:175)
Caused by: com.google.protobuf.RuntimeVersion$ProtobufRuntimeVersionException: Detected incompatible Protobuf Gencode/Runtime versions when loading com.google.devtools.build.android.proto.SerializeFormat$Header: gencode 4.29.3, runtime 4.29.0. Runtime version cannot be older than the linked gencode version.
	at com.google.protobuf.RuntimeVersion.validateProtobufGencodeVersionImpl(RuntimeVersion.java:117)
	at com.google.protobuf.RuntimeVersion.validateProtobufGencodeVersion(RuntimeVersion.java:71)
	at com.google.devtools.build.android.proto.SerializeFormat$Header.<clinit>(SerializeFormat.java:86)
	... 5 more
Target //tests/unit/aar_import:starlark_aar_import_test failed to build

I'm not entirely sure why these versions are getting mixed, since things for rules_android should be coming from rules_android

[ahumesky]

Updating rules_android to 29.3 and testing locally seems to get the tests working again, so we'll need a release of rules_android as well

[ahumesky]

Updated the PR to temporarily use a specific commit of rules_android to test protobuf 29.3 ... and windows tests are failing again...

[ahumesky]

We've released rules_android 0.6.2 for aligning the protobuf versions to 29.3, and I've updated this PR to use rules_android 0.6.2

Now we just need a release of rules_kotlin with bazelbuild/rules_kotlin#1274 (@Bencodes @restingbull) and to fix the windows tests (again)

[Bencodes]

We've released rules_android 0.6.2 for aligning the protobuf versions to 29.3, and I've updated this PR to use rules_android 0.6.2

Now we just need a release of rules_kotlin with bazelbuild/rules_kotlin#1274 (@Bencodes @restingbull) and to fix the windows tests (again)

@ahumesky I just cut a new release for you to use bazelbuild/bazel-central-registry#3893

[ahumesky]

Well I found the problem causing the windows tests to fail:
https://github.com/bazelbuild/rules_android/blob/4269050ab06041b0ca0ffb491ceff021c273601f/tools/android/aar_embedded_jars_extractor.py#L103-L105

output_dir and build_target are swapped in the call to main:
https://github.com/bazelbuild/rules_android/blob/4269050ab06041b0ca0ffb491ceff021c273601f/tools/android/aar_embedded_jars_extractor.py#L78-L79

Not sure at the moment how that happened... or why it didn't break just about everything... or why it's only failing on windows.... but at least it's an easy fix. This will require another release of rules_android.

[ahumesky]

Not sure at the moment why it's only failing on windows.... but at least it's an easy fix. This will require another release of rules_android.

This was broken only on windows because the error was in a windows-only codepath. That's fixed now, and we just need now the new version of rules_kotlin to be available on BCR. All the tests are (finally) passing with the new version of rules_android and rules_kotlin (temporarily pulled in through git_override in the module file)

[ahumesky]

Well, rules_kotlin 2.1.3 is now on the BCR, and updating to that and merging the latest commits in rules_jvm_external reveals a small golden test breakage which is easy to fix... but, incredibly, yet more windows failures:

https://buildkite.com/bazel/rules-jvm-external-examples/builds/4664#0195ca80-5c92-43f3-83f6-47cd5365b6b7

and it seems to be the same issue that I already submitted fixes for:

bazelbuild/rules_kotlin@4e3b18e

[ahumesky]

Thankfully it just looks like the example needs its MODULE.bazel file updated to use the latest rules_kotlin:

rules_jvm_external/examples/kt_jvm_export/MODULE.bazel

Line 3 in d8af221

bazel_dep(name = "rules_kotlin", version = "2.1.0")

[ahumesky]

Well, updating the example to rules_kotlin 2.1.3 didn't seem to work, which is a bit worrying

[ahumesky]

The bazel build log says it was using 2.1.3 anyway because of the version from rules_jvm_external:

https://buildkite.com/bazel/rules-jvm-external-examples/builds/4664#0195ca80-5c92-43f3-83f6-47cd5365b6b7/275-287

[ahumesky]

Ok, I see the problem now... if I use the source code zip for the 2.1.3 rule_kotlin release, it works, but that's not actually the release, there's a release tar from rules_kotlin's release process, and that does not work. The build file for the JdepsMerge that actually gets used for the release is over here:
https://github.com/bazelbuild/rules_kotlin/blob/59a92533a7d3fb8950951c4ac9acbe4d353e273f/src/main/kotlin/BUILD.release.bazel#L77-L85

and that needs the equivalent to the change I made here:
bazelbuild/rules_kotlin@4e3b18e#diff-0027a8575d8ad08c2f95dd811d45114e8228563a5b45e58db4110a1abd62f7aaR71

Unpacking the release and using local_path_override and adding this to jdeps_merger in src/main/kotlin/BUILD file got it to work locally on windows:

data = [
  "@com_github_jetbrains_kotlin//:annotations",
  "@com_github_jetbrains_kotlin//:kotlin-stdlib",
  "@com_github_jetbrains_kotlin//:kotlin-stdlib-jdk7",
  "@com_github_jetbrains_kotlin//:kotlin-stdlib-jdk8",
  "@com_github_jetbrains_kotlin//:kotlinx-coroutines-core-jvm",
  "@com_github_jetbrains_kotlin//:trove4j",
]

So we'll need another rules_kotlin release....

[ahumesky]

I've opened a PR for the fixes described in the above comment: bazelbuild/rules_kotlin#1324

[ahumesky]

I've updated the PR to the latest rules_kotlin which includes bazelbuild/rules_kotlin#1324, and now all the tests are passing. I'll take this out of draft

[shs96c]

@ahumesky, I'm sorry I missed your update. Could you please rebase, and I'll merge this ASAP.

[ahumesky]

Ok in the course of rebasing I managed to accidentally blow away all my changes and automatically close the PR, but fixed now -- thanks for taking a look!

[ahumesky]

Fixed the tests after rebase