Add krunkit driver supporting GPU acceleration on macOS #20826

nirs · 2025-05-23T19:11:24Z

krunkit is a tool to launch configurable virtual machines using the libkrun platform, optimized for GPU accelerated virtual machines and AI workloads on Apple silicon.

It is mostly compatible with vfkit; the driver is a simplified copy of the vfkit driver.

To experiment with the krunkit driver see https://github.com/medyagh/ai-playground-minikube/tree/main/macos

Benchmarking GPU compute workloads shows 2 orders of magnitude improvement with Virtio-GPU, and fully utilized GPU:

Status

Tested using locally built iso
Start/delete clusters
GPU available in the guest
Run GPU benchmarks
Test unsupported platform (linux/amd64)
Test krunkit is not installed
Test vment-helper not installed
Test vment-helper not configured

Integration test results

Created with iso from #21089 using:

env TEST_ARGS="-minikube-start-args='--driver=krunkit --iso-url=file:///Users/nir/src/minikube/minikube-arm64-minimal-k8s.iso'" make integration

krunkit.tar.gz

Fixes #20803

k8s-ci-robot · 2025-05-23T19:11:34Z

Hi @nirs. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

minikube-bot · 2025-05-23T19:12:02Z

Can one of the admins verify this patch?

nirs · 2025-05-25T13:01:51Z

@afbjorklund can you review this?

I think the issue of not having /dev/dri is using too old kernel. We are using 5.10.207 while libkrun seems to require 5.16 or later.

libkrun for macOS is using venus, which requires

The Venus driver requires supports for

VIRTGPU_PARAM_3D_FEATURES
VIRTGPU_PARAM_CAPSET_QUERY_FIX
VIRTGPU_PARAM_RESOURCE_BLOB
VIRTGPU_PARAM_HOST_VISIBLE
VIRTGPU_PARAM_CROSS_DEVICE
VIRTGPU_PARAM_CONTEXT_INIT

from the virtio-gpu kernel driver, unless vtest is used. That usually means the guest kernel should be at least 5.16

So it seems that we need to move to newer kernel. Do you know why are stuck with 5.10?

minikube/deploy/iso/minikube-iso/configs/minikube_aarch64_defconfig

Line 38 in ec2b7c1

BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE="5.10.207"

afbjorklund · 2025-05-25T14:07:53Z

So it seems that we need to move to newer kernel. Do you know why are stuck with 5.10?

I don't think it is stuck with anything, just that it was using the LTS versions...

It seems that a new kernel version was not included, in the minikube OS upgrade.

Kicbase/ISO: Update buildroot from 2023.02.9 to 2025.2 #20720

Buildroot supports many: https://github.com/buildroot/buildroot/blob/2025.02.x/linux/linux.hash

5.4
5.10
5.15
6.1
6.6
6.12

See also https://www.kernel.org/ ("longterm")

nirs · 2025-05-25T14:21:56Z

So it seems that we need to move to newer kernel. Do you know why are stuck with 5.10?

I don't think it is stuck with anything, just that it was using the LTS versions...

It seems that a new kernel version was not included, in the minikube OS upgrade.

Kicbase/ISO: Update buildroot from 2023.02.9 to 2025.2 #20720

Buildroot supports many: https://github.com/buildroot/buildroot/blob/2025.02.x/linux/linux.hash

5.4

5.10

5.15

6.1

6.6

6.12

See also https://www.kernel.org/ ("longterm")

So should we try to update the kernel to latest longterm version (6.12.30)?

afbjorklund · 2025-05-25T14:23:01Z

There was some talk about bumping to a newer kernel: (6.6?)

ISO upgrades to support user namespaces #18488 (comment)

But that was last year, and for the 2024.02.x OS. So maybe.

It seems that it is only needed for this AI feature, though?

nirs · 2025-05-25T17:52:11Z

It seems that it is only needed for this AI feature, though?

Yes, this is for AI on Apple silicon. I can try to get build an iso to play with it, and when we a working iso we can discuss how to proceed with the upgrade.

afbjorklund · 2025-05-25T18:07:55Z

So you could still use vfkit for all other (non-AI) usage of Kubernetes.

Or maybe even make it a single driver, if the syntax is close enough...

pkg/drivers/krunkit/krunkit.go

nirs · 2025-07-19T17:58:29Z

@medyagh all comments addressed. Keeping as draft since it is rebased on #21089, required for vfkit and krunkit.

nirs · 2025-07-19T19:38:05Z

Rebased on #21089 to consume the new iso.

pkg/minikube/driver/driver.go

site/content/en/docs/drivers/krunkit.md

medyagh

thank you @nirs just two small minor typos and then we can merge this

krunkit is a tool to launch configurable virtual machines using the libkrun platform, optimized for GPU accelerated virtual machines and AI workloads on Apple silicon. It is mostly compatible with vfkit; the driver is a simplified copy of the vfkit driver. Unlike vfkit, krunkit is available only on Apple silicon. Changes compared to vfkit driver: - krunkit requires unix socket for networking, so we must use vment-helper. - krunkit does not support HardStop, so we kill it using SIGKILL. - We must enable vmnet offloading, required for krunkit. - The code was simplified since vmnet-helper is always used - Code was cleaned up to use .ResolveStorePath() - Unused Upgrade() function was removed - Types and functions that should not be public made private We require krunkit 0.2.2, supporting --restul-uri=unix://.

Previously it was used only for vfkit, so we suggested to fallback to the `nat` network. This advice is not relevant to krunkit or to qemu (which can also use vmnet-helper). Change the error to recommend installing vment-helper. We need to think how we can recommend other networks for vfkit and qemu. Another solution is to create error for every driver+network combination but this seems hard to manage.

This is the same way that we test vfkit. This test is not running in the CI. Issues: - Need to install and configure vment-helper (requires root).

nirs · 2025-07-20T02:13:24Z

thank you @nirs just two small minor typos and then we can merge this

Fixed

nirs · 2025-07-20T02:30:34Z

Testing krunkit with unsupported platform

$ out/minikube start --driver krunkit
😄  minikube v1.36.0 on Fedora 42 (kvm/amd64)

❌  Exiting due to DRV_UNSUPPORTED_OS: The driver 'krunkit' is not supported on linux/amd64

Testing krunkit not installed

% out/minikube start --driver krunkit
😄  minikube v1.36.0 on Darwin 15.5 (arm64)
✨  Using the krunkit (experimental) driver based on user configuration

🤷  Exiting due to PROVIDER_KRUNKIT_NOT_FOUND: The 'krunkit' provider was not found: exec: "krunkit": executable file not found in $PATH
💡  Suggestion: Run 'brew tap slp/krunkit && brew install krunkit'
📘  Documentation: https://minikube.sigs.k8s.io/docs/reference/drivers/krunkit/

Testing vmnet-helper not installed

% out/minikube start --driver krunkit
😄  minikube v1.36.0 on Darwin 15.5 (arm64)
✨  Using the krunkit (experimental) driver based on user configuration

🤷  Exiting due to PROVIDER_KRUNKIT_NOT_FOUND: The 'krunkit' provider was not found: stat /opt/vmnet-helper/bin/vmnet-helper: no such file or directory
💡  Suggestion: Install and configure vment-helper
📘  Documentation: https://minikube.sigs.k8s.io/docs/reference/drivers/krunkit/

Testing vment-helper not configured

sudores rule not installed, password required during startup.

% out/minikube start --driver krunkit
😄  minikube v1.36.0 on Darwin 15.5 (arm64)
💡  Unable to run vmnet-helper without a password
    To configure vment-helper to run without a password, please check the documentation:
    https://github.com/nirs/vmnet-helper/#granting-permission-to-run-vmnet-helper
Password:
✨  Using the krunkit (experimental) driver based on user configuration
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🔥  Creating krunkit VM (CPUs=2, Memory=6144MB, Disk=20000MB) ...
🐳  Preparing Kubernetes v1.33.2 on Docker 28.3.2 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

Testing vment-helper configured

Using passwordless sudo.

% sudo --reset-timestamp

% out/minikube start --driver krunkit                                                                 
😄  minikube v1.36.0 on Darwin 15.5 (arm64)
✨  Using the krunkit (experimental) driver based on user configuration
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🔥  Creating krunkit VM (CPUs=2, Memory=6144MB, Disk=20000MB) ...
🐳  Preparing Kubernetes v1.33.2 on Docker 28.3.2 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

minikube-pr-bot · 2025-07-20T03:44:57Z

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 20826) |
+----------------+----------+---------------------+
| minikube start | 49.3s    | 52.2s               |
| enable ingress | 14.8s    | 14.9s               |
+----------------+----------+---------------------+

Times for minikube start: 48.7s 52.6s 50.3s 48.1s 46.7s
Times for minikube (PR 20826) start: 50.6s 54.4s 50.4s 55.5s 50.3s

Times for minikube (PR 20826) ingress: 14.9s 15.9s 14.9s 14.9s 13.9s
Times for minikube ingress: 14.6s 14.5s 14.9s 15.0s 14.9s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 20826) |
+----------------+----------+---------------------+
| minikube start | 22.7s    | 22.9s               |
| enable ingress | 12.9s    | 13.1s               |
+----------------+----------+---------------------+

Times for minikube start: 21.2s 22.3s 22.8s 22.5s 24.6s
Times for minikube (PR 20826) start: 21.8s 23.6s 22.8s 22.6s 23.7s

Times for minikube ingress: 12.7s 12.2s 12.7s 13.2s 13.3s
Times for minikube (PR 20826) ingress: 12.3s 12.8s 13.8s 12.8s 13.8s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 20826) |
+----------------+----------+---------------------+
| minikube start | 22.4s    | 22.5s               |
| enable ingress | 26.0s    | 26.4s               |
+----------------+----------+---------------------+

Times for minikube start: 20.5s 24.4s 21.8s 24.9s 20.2s
Times for minikube (PR 20826) start: 24.6s 21.9s 20.7s 20.6s 24.7s

Times for minikube ingress: 22.7s 38.7s 22.7s 22.7s 23.2s
Times for minikube (PR 20826) ingress: 22.8s 40.2s 22.7s 22.8s 23.2s

medyagh

Excellent work @nirs this is gonna be the highlight of the next release and enables developing AI/ML on kubernetes macbook machines !

k8s-ci-robot · 2025-07-21T19:59:04Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: medyagh, nirs

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [medyagh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels May 23, 2025

k8s-ci-robot requested a review from ComradeProgrammer May 23, 2025 19:11

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label May 23, 2025

k8s-ci-robot requested a review from spowelljr May 23, 2025 19:11

k8s-ci-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label May 23, 2025

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 24, 2025

nirs force-pushed the krunkit branch from d5c40d6 to cb9d010 Compare May 24, 2025 21:15

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 24, 2025

nirs force-pushed the krunkit branch 3 times, most recently from 7cda3a5 to cbe0012 Compare May 25, 2025 01:25

nirs force-pushed the krunkit branch from cbe0012 to de02e9e Compare May 25, 2025 13:58

nirs mentioned this pull request May 25, 2025

clean up duplicate code in multiple files #20828

Open

nirs force-pushed the krunkit branch from de02e9e to 4afebdd Compare May 25, 2025 21:33

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels May 25, 2025

nirs force-pushed the krunkit branch from 4afebdd to 7d04f73 Compare May 25, 2025 22:20

nirs commented May 26, 2025

View reviewed changes

pkg/drivers/krunkit/krunkit.go Outdated Show resolved Hide resolved

nirs commented May 26, 2025

View reviewed changes

pkg/drivers/krunkit/krunkit.go Outdated Show resolved Hide resolved

nirs commented May 26, 2025

View reviewed changes

pkg/drivers/krunkit/krunkit.go Show resolved Hide resolved

nirs commented May 26, 2025

View reviewed changes

pkg/drivers/krunkit/krunkit.go Outdated Show resolved Hide resolved

This comment has been minimized.

Sign in to view

nirs mentioned this pull request Jul 19, 2025

cleanup: Driver.generateDiskImage() duplicated in multiple drivers #21090

Open

nirs force-pushed the krunkit branch from 8329933 to d06be70 Compare July 19, 2025 17:55

This comment has been minimized.

Sign in to view

nirs force-pushed the krunkit branch from d06be70 to 768f91a Compare July 19, 2025 19:37

This comment has been minimized.

Sign in to view

medyagh reviewed Jul 20, 2025

View reviewed changes

pkg/minikube/driver/driver.go Outdated Show resolved Hide resolved

medyagh reviewed Jul 20, 2025

View reviewed changes

site/content/en/docs/drivers/krunkit.md Outdated Show resolved Hide resolved

medyagh requested changes Jul 20, 2025

View reviewed changes

nirs force-pushed the krunkit branch from 768f91a to caa20a0 Compare July 20, 2025 02:05

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jul 20, 2025

nirs added 4 commits July 20, 2025 05:11

hack: Add krunkit integration test

cff8099

This is the same way that we test vfkit. This test is not running in the CI. Issues: - Need to install and configure vment-helper (requires root).

site: Add krunkit driver documentation

8ec2c07

nirs force-pushed the krunkit branch from caa20a0 to 8ec2c07 Compare July 20, 2025 02:12

nirs marked this pull request as ready for review July 20, 2025 02:13

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 20, 2025

k8s-ci-robot requested a review from medyagh July 20, 2025 02:13

medyagh approved these changes Jul 21, 2025

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 21, 2025

medyagh merged commit 56975bc into kubernetes:master Jul 21, 2025
27 of 36 checks passed

nirs deleted the krunkit branch July 21, 2025 20:00

Add krunkit driver supporting GPU acceleration on macOS #20826

Add krunkit driver supporting GPU acceleration on macOS #20826

Conversation

nirs commented May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Status

Integration test results

Uh oh!

k8s-ci-robot commented May 23, 2025

Uh oh!

minikube-bot commented May 23, 2025

Uh oh!

nirs commented May 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

afbjorklund commented May 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nirs commented May 25, 2025

Uh oh!

afbjorklund commented May 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nirs commented May 25, 2025

Uh oh!

afbjorklund commented May 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

nirs commented Jul 19, 2025

Uh oh!

This comment has been minimized.

nirs commented Jul 19, 2025

Uh oh!

This comment has been minimized.

Uh oh!

Uh oh!

medyagh left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nirs commented Jul 20, 2025

Uh oh!

nirs commented Jul 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing krunkit with unsupported platform

Testing krunkit not installed

Testing vmnet-helper not installed

Testing vment-helper not configured

Testing vment-helper configured

Uh oh!

minikube-pr-bot commented Jul 20, 2025

Uh oh!

medyagh left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

k8s-ci-robot commented Jul 21, 2025

Uh oh!

Uh oh!

Uh oh!

nirs commented May 23, 2025 •

edited

Loading

nirs commented May 25, 2025 •

edited

Loading

afbjorklund commented May 25, 2025 •

edited

Loading

afbjorklund commented May 25, 2025 •

edited

Loading

afbjorklund commented May 25, 2025 •

edited

Loading

medyagh left a comment •

edited

Loading

nirs commented Jul 20, 2025 •

edited

Loading

medyagh left a comment •

edited

Loading