Skip to content

Add initial support for Kotlin Native toolchain and klib compilation #1351

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 10 commits into
base: master
Choose a base branch
from

Conversation

smocherla-brex
Copy link

@smocherla-brex smocherla-brex commented Jul 13, 2025

In #1347 (comment), it was mentioned that it's preferred to bootstrap a Kotlin native toolchain first to leverage the new IR compiler to utilize a compilation model to output klibs in a generic way that can be leveraged for JS and potentially other platforms in the future. This PR tries to add that to the toolchain and also adds a new rule kt_klib_library rule which is intended to use the Kotlin-native compiler to produce klibs for platform indepedent code that can be shared between various platform specific targets (think of commonMain in a gradle project) and can be in the future consumed by platform specific JS/JVM (and maybe WASM targets as well).

Some notes on the implementation details:

  1. I had to write a repository rule kotlin_native_compiler_repository to bootstrap/download the native distributions for different platforms supported along with its "capabilities" repository similar to how kotlinc is setup. I'll add some more comments inline.
  2. Make sure to expose konan.home or the native compiler files as a TreeArtifact/directory to avoid passing a massive number of files during compilation.
  3. Pass the relevant flags during klib compilation with its own task executor with key ones being produce=library
  4. Additionally, expose alias targets like //kotlin/compiler:kotlin-native which is supposed to alias to the native compiler jar/distribution on the execution platform.
  5. Add examples for simple one (using stdlib) and one with deps (kt_klib_library depending on another target). Add some junit test cases to assert klib has some entries we expect.
  6. Add a few starlark unit test cases.

A simple example shown below:

smocherla@NLC2L54QQY rules_kotlin_oss % bazel build //src/test/data/klib:deps_main
WARNING: For repository 'com_google_protobuf', the root module requires module version protobuf@29.0, but got protobuf@29.3 in the resolved dependency graph.
WARNING: For repository 'rules_python', the root module requires module version rules_python@0.23.1, but got rules_python@0.37.2 in the resolved dependency graph.
INFO: Analyzed target //src/test/data/klib:deps_main (280 packages loaded, 33903 targets configured).

[1,205 / 1,207] Compiling Kotlin to Klib //src/test/data/klib:deps_greeting { kt: 1 }; 3s worker
Target //src/test/data/klib:deps_main up-to-date:
  bazel-bin/src/test/data/klib/deps_main.klib
INFO: Elapsed time: 110.427s, Critical Path: 39.07s
INFO: 1207 processes: 582 internal, 604 darwin-sandbox, 

TODO:

  • Fix some of the integration test failures
  • Decide if a separate rule is needed for klibs or fit it into existing platform-specific rules.

@@ -19,6 +19,8 @@ bazel_dep(name = "rules_shell", version = "0.4.1")

bazel_dep(name = "buildifier_prebuilt", version = "8.0.3", dev_dependency = True)

bazel_dep(name = "aspect_bazel_lib", version = "2.19.4")
Copy link
Author

@smocherla-brex smocherla-brex Jul 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added this to leverage copy_to_directory to expose the Kotlin native compiler distribution as a directory in the toolchain here (as it's massive and has a ton of files)

),
]

kt_klib_library = rule(
Copy link
Author

@smocherla-brex smocherla-brex Jul 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the new kt_klib_library rule that produces klibs using the kotlin/native toolchain. Potentially we could just have it as action and have an attr on the existing rules to trigger klib compilation on them as well (I'm not sure if that's better - let me know)

// within the current working directory to isolate its cache (which is unique to this worker anyway)
// Ideally we disable caching though and rely only on Bazel
add("-Xauto-cache-dir=${autoCacheDirectory.absolutePathString()}")
add("-Xauto-cache-from=${autoCacheFromDirectory.absolutePathString()}")
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without this, the compiler tries to write within the cache in the external repository where konan.home is and bazel errors out. Ideally we disable caching but there doesn't seem to be a flag for this.

val konanHome = System.getenv("KONAN_HOME")
requireNotNull(konanHome) {"KONAN_HOME env var must be set!"}

System.setProperty("konan.home", konanHome)
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is needed for the compiler to find the native code/distribution to produce klib.

@@ -0,0 +1,98 @@
load("//src/main/starlark/core/repositories/kotlin:templates.bzl", "TEMPLATES")

def _kotlin_capabilities_impl(repository_ctx):
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was copied into a separate file to allow leveraging it with kotlin_native_compiler_repository

out = "klib/cache/macos_arm64STATIC/klib_cache_marker",
content = [
"Marker file intended to create the klib system cache placeholder even if we don't use it. The native compiler errors out otherwise",
"See See https://github.com/JetBrains/kotlin/blob/v2.1.21/kotlin-native/backend.native/compiler/ir/backend.native/src/org/jetbrains/kotlin/backend/konan/KonanConfig.kt#L567"
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The compiler on bootstrap tries to create klib/cache/<platform>STATIC directory in the external repo and this is non-hermetic and causes Bazel to fail. To workaround it (unfortunately there doesn't appear to be a way to disable it with a flag), so we create this marker/dummy file to avoid running into the issue (happy to consider other solutions)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant