vfkit: Bump to Preferred driver on macOs #20808

nirs · 2025-05-21T00:28:16Z

vfkit is using the native virtualization framework, provides good best performance and all the features needed by minikube. It is also well maintained and used by other projects like podman.

This fixes the automatic driver selection. When we start minikube on a system with both vfkit and qemu, vfkit is selected:

% minikube start
😄  minikube v1.35.0 on Darwin 15.5 (arm64)
✨  Automatically selected the vfkit driver. Other choices: qemu2, ssh, podman (experimental)
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🔥  Creating vfkit VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
🐳  Preparing Kubernetes v1.33.1 on Docker 28.0.4 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

Fixes #20807

vfkit is using the native virtualization framework, provides good best performance and all the features needed by minikube. It is also well maintained and used by other projects like podman. This fixes the automatic driver selection. When we start minikube on a system with both vfkit and qemu, vfkit is selected: % minikube start 😄 minikube v1.35.0 on Darwin 15.5 (arm64) ✨ Automatically selected the vfkit driver. Other choices: qemu2, ssh, podman (experimental) 👍 Starting "minikube" primary control-plane node in "minikube" cluster 🔥 Creating vfkit VM (CPUs=2, Memory=6000MB, Disk=20000MB) ... 🐳 Preparing Kubernetes v1.33.1 on Docker 28.0.4 ... ▪ Generating certificates and keys ... ▪ Booting up control plane ... ▪ Configuring RBAC rules ... 🔗 Configuring bridge CNI (Container Networking Interface) ... 🔎 Verifying Kubernetes components... ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5 🌟 Enabled addons: default-storageclass, storage-provisioner 🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

k8s-ci-robot · 2025-05-21T00:28:25Z

Hi @nirs. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

minikube-bot · 2025-05-21T00:30:03Z

Can one of the admins verify this patch?

nirs · 2025-05-21T00:40:35Z

@afbjorklund what do you think?

medyagh · 2025-05-21T01:00:02Z

pkg/minikube/registry/drvs/vfkit/vfkit.go

@@ -41,13 +42,17 @@ const (
 )

 func init() {
+	priority := registry.Unknown
+	if runtime.GOOS == "darwin" {
+		priority = registry.Preferred


I think is better than preffered registry.Default so it be the same level as qemu,

based on comment from @nirs I agree to bump prefered.

Both vfkit and qemu are using the native hypervisor interface (hypervisor framewlrk). But vfkit is using Apple virtualization framework maintained by the vendor. qemu is not well maintained on macOS. You may not notice it because you was using an old mac, but M4 is still not fully supported in qemu. Not long ago starting qemu on M4 was crashing and it took few month until they fixed the issue, only after I pushed for it and convinced UTM maintainer to send a trivial workaround to avoid the crash to qemu. lima is using VZ as the default vm type instead of qemu since 1.0 (Nov 6, 2024). Podman switch to vfkit long time ago (was using qemu before). We should move forward and recommend vfkit. Unless we have some important features that does not work on vfkit and work on qemu. I'm not a ware of any.

cfergeau

Looks good to me!

cfergeau · 2025-05-21T07:54:49Z

pkg/minikube/registry/drvs/vfkit/vfkit.go

@@ -104,7 +109,7 @@ func configure(cfg config.ClusterConfig, n config.Node) (interface{}, error) {
 func status() registry.State {
 	_, err := exec.LookPath("vfkit")
 	if err != nil {
-		return registry.State{Error: err, Fix: "Run 'brew tap cfergeau/crc && brew install vfkit'", Doc: docURL}
+		return registry.State{Error: err, Fix: "Run 'brew install vfkit'", Doc: docURL}


👍
vfkit in cfergeau/crc will no longer be updated now that it’s in the main repo. I should remove it from this personal tap.

From crc-org/vfkit@1228143

medyagh · 2025-05-21T16:50:14Z

pkg/minikube/registry/drvs/vfkit/vfkit.go

@@ -41,13 +42,17 @@ const (
 )

 func init() {
+	priority := registry.Unknown
+	if runtime.GOOS == "darwin" {
+		priority = registry.Preferred


based on comment from @nirs I agree to bump prefered.

Both vfkit and qemu are using the native hypervisor interface (hypervisor framewlrk). But vfkit is using Apple virtualization framework maintained by the vendor. qemu is not well maintained on macOS. You may not notice it because you was using an old mac, but M4 is still not fully supported in qemu. Not long ago starting qemu on M4 was crashing and it took few month until they fixed the issue, only after I pushed for it and convinced UTM maintainer to send a trivial workaround to avoid the crash to qemu. lima is using VZ as the default vm type instead of qemu since 1.0 (Nov 6, 2024). Podman switch to vfkit long time ago (was using qemu before). We should move forward and recommend vfkit. Unless we have some important features that does not work on vfkit and work on qemu. I'm not a ware of any.

medyagh · 2025-05-21T16:53:23Z

/ok-to-test

medyagh · 2025-05-21T18:31:50Z

/lgtm

medyagh · 2025-05-21T18:32:29Z

/lgtm

k8s-ci-robot · 2025-05-21T18:32:36Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: medyagh, nirs

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [medyagh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

minikube-pr-bot · 2025-05-21T19:57:06Z

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 20808) |
+----------------+----------+---------------------+
| minikube start | 48.9s    | 50.3s               |
| enable ingress | 14.9s    | 14.8s               |
+----------------+----------+---------------------+

Times for minikube start: 49.8s 49.3s 46.0s 47.8s 51.4s
Times for minikube (PR 20808) start: 52.2s 48.6s 51.5s 51.7s 47.5s

Times for minikube ingress: 15.0s 15.0s 15.0s 14.9s 14.5s
Times for minikube (PR 20808) ingress: 14.5s 14.9s 14.5s 15.0s 15.1s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 20808) |
+----------------+----------+---------------------+
| minikube start | 22.8s    | 24.0s               |
| enable ingress | 12.3s    | 12.1s               |
+----------------+----------+---------------------+

Times for minikube ingress: 12.8s 12.3s 12.8s 12.8s 10.8s
Times for minikube (PR 20808) ingress: 11.8s 10.3s 12.8s 12.8s 12.8s

Times for minikube start: 22.5s 24.3s 22.6s 22.3s 22.3s
Times for minikube (PR 20808) start: 22.5s 24.1s 25.5s 22.4s 25.3s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 20808) |
+----------------+----------+---------------------+
| minikube start | 22.9s    | 22.2s               |
| enable ingress | 34.1s    | 29.3s               |
+----------------+----------+---------------------+

Times for minikube start: 24.3s 25.5s 22.3s 20.6s 21.8s
Times for minikube (PR 20808) start: 24.3s 22.5s 20.2s 20.6s 23.2s

Times for minikube ingress: 31.3s 37.8s 23.8s 38.8s 38.8s
Times for minikube (PR 20808) ingress: 22.8s 22.8s 38.8s 39.3s 22.8s

cfergeau · 2025-05-22T07:15:21Z

Not long ago starting qemu on M4 was crashing and it took few month until they fixed the issue

For the record, this is not an isolated event, there was a similar problem around the time podman switched to vfkit, qemu hang on startup which took several weeks/months to be fixed.

Unless we have some important features that does not work on vfkit and work on qemu. I'm not a ware of any.

vfkit has support for virtiofs and rosetta (important features off the top of my head), and the fact that Apple maintains most of the virtualization code is both a plus and a minus, but in my opinion mostly a plus. It overall works well, we had minimal issues over the years, but when there are issues, we are dependent on Apple to fix them, containers/podman#25121 is a recent example of this. This bug only impacted some Intel macs, and is fixed with macOS 15.5
One big missing thing, but I’m not sure qemu has this either, is gpu acceleration support, which I guess could be useful to minikube (?). We’ll know at WWDC in a few weeks if they are going to add this to macOS 16 :)

nirs · 2025-05-22T18:28:31Z

One big missing thing, but I’m not sure qemu has this either, is gpu acceleration support, which I guess could be useful to minikube (?). We’ll know at WWDC in a few weeks if they are going to add this to macOS 16 :)

I started to work on #20803 for GPU acceleration.

afbjorklund · 2025-05-23T06:20:53Z

The main feature of
QEMU is that it is available on all platforms, something that was previously done by VirtualBox.

Having to split minikube into different hypervisors and different runtimes (not just Docker) complicated things a lot.

afbjorklund · 2025-05-25T07:51:42Z

@afbjorklund what do you think?

~~Previously it was docker that was the Preferred driver, while something like vfkit could be the Default driver.~~

EDIT: no, it was the names that are confusing. Docker is "HighlyPreferred", while others are just "Preferred"

If you wanted to run in a separate VM, instead of running in a container, there was always the --vm option.

nirs · 2025-05-25T13:35:28Z

@afbjorklund what do you think?

~~Previously it was docker that was the Preferred driver, while something like vfkit could be the Default driver.~~

EDIT: no, it was the names that are confusing. Docker is "HighlyPreferred", while others are just "Preferred"

docker on macOS should be Obsolete, it requires so you should really use one of the open source vm drivers.

k8s-ci-robot requested a review from medyagh May 21, 2025 00:28

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label May 21, 2025

k8s-ci-robot requested a review from spowelljr May 21, 2025 00:28

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels May 21, 2025

medyagh requested changes May 21, 2025

View reviewed changes

cfergeau reviewed May 21, 2025

View reviewed changes

medyagh approved these changes May 21, 2025

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 21, 2025

medyagh changed the title ~~vfkit: Change priority to Preferred~~ vfkit: Bump to Preferred driver on macOs May 21, 2025

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels May 21, 2025

k8s-ci-robot assigned medyagh May 21, 2025

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 21, 2025

medyagh merged commit 3496901 into kubernetes:master May 21, 2025
36 of 50 checks passed

nirs deleted the vfkit-priority branch May 22, 2025 11:02

vfkit: Bump to Preferred driver on macOs #20808

vfkit: Bump to Preferred driver on macOs #20808

Conversation

nirs commented May 21, 2025

Uh oh!

k8s-ci-robot commented May 21, 2025

Uh oh!

minikube-bot commented May 21, 2025

Uh oh!

nirs commented May 21, 2025

Uh oh!

medyagh May 21, 2025

Choose a reason for hiding this comment

Uh oh!

medyagh May 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cfergeau left a comment

Choose a reason for hiding this comment

Uh oh!

cfergeau May 21, 2025

Choose a reason for hiding this comment

Uh oh!

medyagh May 21, 2025

Choose a reason for hiding this comment

Uh oh!

afbjorklund May 25, 2025

Choose a reason for hiding this comment

Uh oh!

medyagh May 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

medyagh commented May 21, 2025

Uh oh!

medyagh commented May 21, 2025

Uh oh!

medyagh commented May 21, 2025

Uh oh!

k8s-ci-robot commented May 21, 2025

Uh oh!

Uh oh!

minikube-pr-bot commented May 21, 2025

Uh oh!

cfergeau commented May 22, 2025

Uh oh!

nirs commented May 22, 2025

Uh oh!

afbjorklund commented May 23, 2025

Uh oh!

afbjorklund commented May 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nirs commented May 25, 2025

Uh oh!

Uh oh!

medyagh May 21, 2025 •

edited

Loading

medyagh May 21, 2025 •

edited

Loading

afbjorklund commented May 25, 2025 •

edited

Loading