Inlining code in Scala targets

[gergelyfabian]

With Scala 2.13.16 if you try inlining code that has been added to your classpath as an ijar/hjar it throws an exception:

error: java.lang.IndexOutOfBoundsException
	at scala.tools.asm.tree.InsnList.get(InsnList.java:94)
	at scala.tools.nsc.backend.jvm.analysis.BackendUtils$.computeMaxLocalsMaxStack(BackendUtils.scala:728)
	at scala.tools.nsc.backend.jvm.analysis.BackendUtils$.maxLocals(BackendUtils.scala:636)
	at scala.tools.nsc.backend.jvm.opt.Inliner.inlineCallsite(Inliner.scala:856)
	at scala.tools.nsc.backend.jvm.opt.Inliner.$anonfun$runInliner$10(Inliner.scala:352)
	at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:619)
	at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:617)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:935)
	at scala.collection.IterableOps$WithFilter.foreach(Iterable.scala:905)
	at scala.tools.nsc.backend.jvm.opt.Inliner.runInliner(Inliner.scala:378)
	at scala.tools.nsc.backend.jvm.opt.Inliner.runInlinerAndClosureOptimizer(Inliner.scala:281)
	at scala.tools.nsc.backend.jvm.PostProcessor.runGlobalOptimizations(PostProcessor.scala:156)
	at scala.tools.nsc.backend.jvm.GeneratedClassHandler$GlobalOptimisingGeneratedClassHandler.complete(GeneratedClassHandler.scala:98)
	at scala.tools.nsc.backend.jvm.GenBCode$BCodePhase.$anonfun$run$1(GenBCode.scala:81)
	at scala.tools.nsc.backend.jvm.GenBCode$BCodePhase.run(GenBCode.scala:78)
	at scala.tools.nsc.Global$Run.compileUnitsInternal(Global.scala:1564)
	at scala.tools.nsc.Global$Run.compileUnits(Global.scala:1548)
	at scala.tools.nsc.Global$Run.compileSources(Global.scala:1540)
	at scala.tools.nsc.Global$Run.compile(Global.scala:1674)
	at scala.tools.nsc.Driver.doCompile(Driver.scala:48)
	at scala.tools.nsc.MainClass.doCompile(Main.scala:30)
	at scala.tools.nsc.Driver.process(Driver.scala:68)
	at io.bazel.rulesscala.scalac.ScalacInvoker.invokeCompiler(ScalacInvoker.java:24)
	at io.bazel.rulesscala.scalac.ScalacWorker.compileScalaSources(ScalacWorker.java:272)
	at io.bazel.rulesscala.scalac.ScalacWorker.work(ScalacWorker.java:88)
	at io.bazel.rulesscala.worker.Worker.persistentWorkerMain(Worker.java:96)
	at io.bazel.rulesscala.worker.Worker.workerMain(Worker.java:49)
	at io.bazel.rulesscala.scalac.ScalacWorker.main(ScalacWorker.java:48)
java.lang.IndexOutOfBoundsException
	at scala.tools.asm.tree.InsnList.get(InsnList.java:94)
	at scala.tools.nsc.backend.jvm.analysis.BackendUtils$.computeMaxLocalsMaxStack(BackendUtils.scala:728)
	at scala.tools.nsc.backend.jvm.analysis.BackendUtils$.maxLocals(BackendUtils.scala:636)
	at scala.tools.nsc.backend.jvm.opt.Inliner.inlineCallsite(Inliner.scala:856)
	at scala.tools.nsc.backend.jvm.opt.Inliner.$anonfun$runInliner$10(Inliner.scala:352)
	at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:619)
	at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:617)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:935)
	at scala.collection.IterableOps$WithFilter.foreach(Iterable.scala:905)
	at scala.tools.nsc.backend.jvm.opt.Inliner.runInliner(Inliner.scala:378)
	at scala.tools.nsc.backend.jvm.opt.Inliner.runInlinerAndClosureOptimizer(Inliner.scala:281)
	at scala.tools.nsc.backend.jvm.PostProcessor.runGlobalOptimizations(PostProcessor.scala:156)
	at scala.tools.nsc.backend.jvm.GeneratedClassHandler$GlobalOptimisingGeneratedClassHandler.complete(GeneratedClassHandler.scala:98)
	at scala.tools.nsc.backend.jvm.GenBCode$BCodePhase.$anonfun$run$1(GenBCode.scala:81)
	at scala.tools.nsc.backend.jvm.GenBCode$BCodePhase.run(GenBCode.scala:78)
	at scala.tools.nsc.Global$Run.compileUnitsInternal(Global.scala:1564)
	at scala.tools.nsc.Global$Run.compileUnits(Global.scala:1548)
	at scala.tools.nsc.Global$Run.compileSources(Global.scala:1540)
	at scala.tools.nsc.Global$Run.compile(Global.scala:1674)
	at scala.tools.nsc.Driver.doCompile(Driver.scala:48)
	at scala.tools.nsc.MainClass.doCompile(Main.scala:30)
	at scala.tools.nsc.Driver.process(Driver.scala:68)
	at io.bazel.rulesscala.scalac.ScalacInvoker.invokeCompiler(ScalacInvoker.java:24)
	at io.bazel.rulesscala.scalac.ScalacWorker.compileScalaSources(ScalacWorker.java:272)
	at io.bazel.rulesscala.scalac.ScalacWorker.work(ScalacWorker.java:88)
	at io.bazel.rulesscala.worker.Worker.persistentWorkerMain(Worker.java:96)
	at io.bazel.rulesscala.worker.Worker.workerMain(Worker.java:49)
	at io.bazel.rulesscala.scalac.ScalacWorker.main(ScalacWorker.java:48)

This seems to be a Scala regression since 2.13.9+ and Scala declares that it will fix it in 2.13.17.

Steps to reproduce:

git clone git@github.com:gergelyfabian/bazel-scala-example.git
cd bazel-scala-example
git checkout scala-compiler-bug
bazel build //example-maven:example-maven

Example code:

package mypackage

import com.google.protobuf.CodedOutputStream

object Maven {
  def repro(): Unit = {
    val result = new Array[Byte](0)
    CodedOutputStream.newInstance(result)
  }
}

Using the following scalacopt: -opt-inline-from:**.

---

Once Scala fixes this bug, it will stop inlining code from signature jar files, but you'll get a warning:

$> scq T.scala -cp libcore-hjar.jar:liblite_runtime_only-hjar.jar '-opt:inline:**' -Wopt:_
T.scala:6: warning: com/google/protobuf/CodedOutputStream::newInstance([B)Lcom/google/protobuf/CodedOutputStream; could not be inlined:
Method com/google/protobuf/CodedOutputStream::newInstance([B)Lcom/google/protobuf/CodedOutputStream; cannot be inlined because it does not have any instructions, even though it is not abstract. The class may come from a signature jar file (such as a Bazel 'hjar').
    CodedOutputStream.newInstance(result)
                                 ^
1 warning

---

This behavior is caused by Bazel using hjars/ijars for dependencies.

---

Related Bazel Slack topics:

• https://bazelbuild.slack.com/archives/CA31HN1T3/p1741940032376929
• https://bazelbuild.slack.com/archives/CA31HN1T3/p1741943837077549

[gergelyfabian]

Scala issue: scala/bug#13097
Scala PR: scala/scala#11016

[gergelyfabian]

Bazel will use ijars for dependencies in your Scala targets for:

• java_library targets
• scala_library targets (when you don't use macros)
• and pretty much all other stuff

Ijars won't be used for:

• external dependencies pulled in with rules_jvm_external (due to some Kotlin incompatibilities)
• scala_macro_library targets
• jars that you have imported with scala_import
• if you enable --nouse_ijars --nojava_header_compilation flags

It happens to be the case that rules_scala already has support for turning off ijars, but it's main purpose is to help with Scala macros (as you need to have the implementation also on the classpath when compiling with macros).

It seems that inlining code is very similar to using macros.

If you want to inline code:

• External dependencies pulled in with rules_jvm_external are fine, for now
• Use scala_macro_library for your in-repo dependencies, that you'd like to inline
• Wrap all java_library targets that you may want to inline with a scala_import, those java_library targets can be autogenerated e.g. for protobuf code
• If nothing else works, you can try disabling ijars/hjars altogether: --nouse_ijars --nojava_header_compilation

Before Scala 2.13.* is fixed, these steps are necessary to make your code compile (when using inlining), after 2.13.17, you need this to ensure Scala will be able to inline that code and won't emit warnings.

[gergelyfabian]

Just for clarity, the inline instruction that my project uses is -opt-inline-from:com.mycompany.**,scalaz.**,scala.**, and then some of the com.mycompany classes are compiled by java_library targets (others are either rules_jvm_external-generated dependencies or scala_macro_library targets, those are OK).

[gergelyfabian]

The Scala bugfix was merged, should be in Scala 2.13.17. This means that Scala compiler won't throw an exception starting from then, but inlining code from hjar/ijar dependencies will be simply ignored.

[SethTisue]

Note that you can test it in a nightly if you like (https://docs.scala-lang.org/overviews/core/nightlies.html)

The current nightly is 2.13.17-bin-f5faa03

[gergelyfabian]

Note that you can test it in a nightly if you like (https://docs.scala-lang.org/overviews/core/nightlies.html)

The current nightly is 2.13.17-bin-f5faa03

Thanks @SethTisue, greatly appreciated!

[gergelyfabian]

Thanks @SethTisue, I can confirm that with 2.13.17-bin-f5faa03 the error is gone. I just need to document how one can use a "nightly" Scala version with rules_scala.

[gergelyfabian]

I haven't seen a warning though, as compiling this code.

[gergelyfabian]

How one can use a "nightly" Scala version... Patch file for rules_scala itself.

diff --git a/scala/scala_cross_version.bzl b/scala/scala_cross_version.bzl
index f7d8980..42188f2 100644
--- a/scala/scala_cross_version.bzl
+++ b/scala/scala_cross_version.bzl
@@ -21,6 +21,7 @@ def default_maven_server_urls():
         "https://maven-central.storage-download.googleapis.com/maven2",
         "https://mirror.bazel.build/repo1.maven.org/maven2",
         "https://jcenter.bintray.com",
+        "https://scala-ci.typesafe.com/artifactory/scala-integration",
     ]
 
 def extract_major_version(scala_version):
@@ -28,7 +29,7 @@ def extract_major_version(scala_version):
     return scala_version[:scala_version.find(".", 2)]
 
 def extract_minor_version(scala_version):
-    return scala_version.split(".")[2]
+    return scala_version.split(".")[2].split("-")[0]
 
 def extract_major_version_underscore(scala_version):
     """Return major Scala version with underscore given a full version,
diff --git a/scala/scala_cross_version_select.bzl b/scala/scala_cross_version_select.bzl
index 615b6a7..4695ff7 100644
--- a/scala/scala_cross_version_select.bzl
+++ b/scala/scala_cross_version_select.bzl
@@ -1,5 +1,5 @@
-load(":scala_cross_version.bzl", "version_suffix")
 load("@rules_scala_config//:config.bzl", "SCALA_VERSIONS")
+load(":scala_cross_version.bzl", "version_suffix")
 
 def select_for_scala_version(default = [], **kwargs):
     """
@@ -61,7 +61,7 @@ def _matches_for_version(scala_version, kwargs, default_value):
 def _match_one_arg(scala_version, matcher_scala_version, compare):
     # Some rudimentary version parsing to allow a lexicographical compare later.
     # Works for versions containing numbers only.
-    scala_version = tuple([int(x) for x in scala_version.split(".")])
+    scala_version = tuple([int(x.split("-")[0]) for x in scala_version.split(".")])
     matcher_scala_version = tuple([int(x) for x in matcher_scala_version.split("_")])
 
     # Compare only a part of version – to allow wildcarding.
diff --git a/third_party/repositories/scala_2_13.bzl b/third_party/repositories/scala_2_13.bzl
index c0e7b0a..4952cdf 100644
--- a/third_party/repositories/scala_2_13.bzl
+++ b/third_party/repositories/scala_2_13.bzl
@@ -3,7 +3,7 @@
 Mostly generated and updated by scripts/create_repository.py.
 """
 
-scala_version = "2.13.17"
+scala_version = "2.13.17-bin-f5faa03"
 
 artifacts = {
     "com_github_jnr_jffi_native": {
@@ -184,8 +184,8 @@ artifacts = {
         ],
     },
     "io_bazel_rules_scala_scala_compiler": {
-        "artifact": "org.scala-lang:scala-compiler:2.13.16",
-        "sha256": "f59982714591e321ba9c087af2c8666e2f5fb92b11a0cef72c2c5e9b342152d3",
+        "artifact": "org.scala-lang:scala-compiler:2.13.17-bin-f5faa03",
+        "sha256": "eae22a432cac9c329f2dd0212cd68f92c8579d87679c8ca1708aee3c57e2d572",
         "deps": [
             "@io_bazel_rules_scala_scala_library",
             "@io_bazel_rules_scala_scala_reflect",
@@ -194,8 +194,8 @@ artifacts = {
         ],
     },
     "io_bazel_rules_scala_scala_library": {
-        "artifact": "org.scala-lang:scala-library:2.13.16",
-        "sha256": "1ebb2b6f9e4eb4022497c19b1e1e825019c08514f962aaac197145f88ed730f1",
+        "artifact": "org.scala-lang:scala-library:2.13.17-bin-f5faa03",
+        "sha256": "d01f578049ccad2a39d12c412632ed6ae41987b6d043865aa57b4dc4fd2b8b4c",
     },
     "io_bazel_rules_scala_scala_parallel_collections": {
         "artifact": "org.scala-lang.modules:scala-parallel-collections_2.13:1.2.0",
@@ -212,8 +212,8 @@ artifacts = {
         ],
     },
     "io_bazel_rules_scala_scala_reflect": {
-        "artifact": "org.scala-lang:scala-reflect:2.13.16",
-        "sha256": "fb49ccd9cac7464486ab993cda20a3c1569d8ef26f052e897577ad2a4970fb1d",
+        "artifact": "org.scala-lang:scala-reflect:2.13.17-bin-f5faa03",
+        "sha256": "9152c89795d6341040abacae69418d49237ba5d984998feb76e371703ba9997f",
         "deps": [
             "@io_bazel_rules_scala_scala_library",
         ],
@@ -439,8 +439,8 @@ artifacts = {
         ],
     },
     "org_scala_lang_scalap": {
-        "artifact": "org.scala-lang:scalap:2.13.16",
-        "sha256": "7963c72c4c74d52278e42b0108ae8ae866d4d1c4579e20209a2f9617e6aacfca",
+        "artifact": "org.scala-lang:scalap:2.13.17-bin-f5faa03",
+        "sha256": "cc322cbd07722d203fd39f8055d1b2ac82420079768457f3f7d98387f7492fcb",
         "deps": [
             "@io_bazel_rules_scala_scala_compiler",
         ],

Apart from that, obviously, you also need to change your project to use this nightly version.

[gergelyfabian]

It happens to be the case that rules_scala already has support for turning off ijars, but it's main purpose is to help with Scala macros (as you need to have the implementation also on the classpath when compiling with macros).

It seems that inlining code is very similar to using macros.

If you want to inline code:

• External dependencies pulled in with rules_jvm_external are fine, for now
• Use scala_macro_library for your in-repo dependencies, that you'd like to inline
• Wrap all java_library targets that you may want to inline with a scala_import, those java_library targets can be autogenerated e.g. for protobuf code
• If nothing else works, you can try disabling ijars/hjars altogether: --nouse_ijars --nojava_header_compilation

Regarding in-repo dependencies.

I think using scala_macro_library is not a so legit solution, as we are using something that wasn't intended for this purpose (and so has side effects).

But, it also happens that scala_macro_library changed its behavior since this February (CC: @jadenPete) and now it adds all transitive deps to the compile classpath, thus making strict dependency checks impossible.
I'd need a solution for creating targets that would be used in inlining, but optimally without using --nouse_ijars --nojava_header_compilation.

I see two solutions:

1. I could either exchange the compile phase in an alternative scala_library rule I create with make_scala_library, turn off buildijar there and just leave it on my side
2. Or, I could contribute a scala_inlined_library rule, an alternative version of scala_library, where buildijar is turned off

I have made nr.2 so far, and it does what it should: https://github.com/bazel-contrib/rules_scala/compare/master...gergelyfabian:rules_scala:scala_inlined_library?expand=1

Question is, which option should take?

CC: @liucijus, @simuons

[gergelyfabian]

The proposed solutions could close this issue as they could provide a general way to do this properly with rules_scala.

[gergelyfabian]

Created a PR with the second solution in #1753