udev SR not working after upgrade to XCP-ng 8.3

[n-buck]

Hi

After updating to 8.3, my SR did not work anymore.
I have created it like this:

mkdir /srv/NasDrives

ln -s /dev/sda2 /srv/NasDrives/sda

xe sr-create name-label="My Nas Storage" name-description="Nas Storage" type=udev content-type=disk device-config:location=/srv/NasDrives


xe sr-scan uuid=id

This results in disks with an unrecognized bus type and 0B size.
I am therefore not able to assign this sr to a vm.

Recreating the SR in a similar manor did not help as well.

[stormi]

Hi, and sorry for the late reply.

Has the situation evolved in the last two weeks?

Pinging @Wescoeur and @Nambrok for insight regarding the use of the udev driver.

[stormi]

And pinging @benjamreis regarding pass-through.

[Nambrok]

The udev sm driver is not PCI passthrough, it makes tapdisk directly show the block device through a PV device to the VM. I will take a look to see if there is any changes in 8.3.

[n-buck]

@stormi thank you for the reply.
The situation has not changed (downgraded to 8.2 again).

How would I test this again, is there a patch released, or would I install again from the iso, then update with yum?

[stormi]

No, no patch released that would address this kind of issue. If you can reproduce on a test host, or find a way for us to reproduce, we can try to find the cause.

[n-buck]

Hello Stormi,
Sorry for the late reply.

Since this is my only set-up and I did not have the storage to backup my data, I was somewhat afraid to test it further on this system.

What I did:

1. Run XCP-NG 8.2.1
2. Create a new Storage-Repository:
   2.1 mkdir /srv/NasDrives
   2.2 xe sr-create name-label="My Nas Storage" name-description="Nas Storage" type=udev content-type=disk device-config:location=/srv/NasDrives
   2.3 ln -s /dev/sda2 /srv/NasDrives/sda ...
   2.4 xe sr-scan uuid=664fb621-a084-8c9a-9858-1041fe5f5a67
3. Assign it to a VM
4. Update to XCP-NG 8.3
   4.1 The Storage-Repository has no drives.
   4.2 Recreating a Storage Repository as described in step 2 did not work either
5. Restore XCP-NG 8.2.1

Since this is also not a supported way to pass the drives to TrueNas, I am in the process to upgrade my system to passing the disks with a HBA.

I hope this is of any help, and thank you for an amazing product!

[jeremfg]

I've just faced this exact issue. I've already commented here also.

My use case is a server that doesn't support IOMMU.

By the sound of it, I should be able to get it working if I install 8.2.1 instead, which would be an acceptable temporary workaround for me. All my other nodes are still running 8.2.1 anyway, this was my first foray into 8.3. I'll test out the downgrade in a few days.

EDIT: I've since made the move to 8.2.1. Making the exact same configuration worked on the older release, so there's definitely something broken specific to 8.3.

[stormi]

Using XCP-ng 8.2.1 is a temporary solution, but it will only last while XCP-ng 8.2.1 is supported (well, you can continue to use it after it's EOL, but without security fixes).

The udev sm driver is not PCI passthrough, it makes tapdisk directly show the block device through a PV device to the VM. I will take a look to see if there is any changes in 8.3.

@Nambrok Did you reach a conclusion regarding this issue?

[Nambrok]

@stormi Sorry, I haven't had time to look yet. I will be investigating when I have time.

[stormi]

@jeremfg could you give us the exact commands that you tried on XCP-ng 8.3 and that didn't work?

[blackliner]

Does this mean there is no way to pass through disks with 8.3? (PCI passthrough of an HBA doesn't count)

[jeremfg]

@jeremfg could you give us the exact commands that you tried on XCP-ng 8.3 and that didn't work?

@stormi
Sorry I didn't see your message until now.
If you are willing to read some bash, this is the exact script I've been running to create the SR.

If you need to see how I use this code, you can find the reference here in function nas_storage_update.

[jeremfg]

Does this mean there is no way to pass through disks with 8.3? (PCI passthrough of an HBA doesn't count)

@blackliner
Not sure how you define "disk" here. You can still for example create a LVM SR, create virtual disks on top of it, and attach those to a VM for example. The regular stuff still works.

But a whole disk as a udev? Nope, doesn't seem to work anymore. The simlinks to /dev/sd[a,b,c] don't show up in the SR. I had to revert back to the latest 8.2.1 for now.

[stormi]

So, @Nambrok will provide details if needed, but here's the situation I understood from talking with him:

• The udev SR driver was never made for what users do with it, that is, assign a whole disk to a VM without doing pass-through. If I'm not mistaken, the udev driver is here mostly to handle removable devices. To my knowledge, this use of the udev driver to attach whole disks or partitions to VMs isn't documented in XCP-ng's official documentation either.
• It was even less designed to assign a partition to a VM, and there even was a check to prevent it, but the check was easily fooled by renaming the symlink.
• Upstream developers have improved the behaviour of the driver so that it would actually follow symlinks, rather than assuming that their name conveys a meaningful information. This makes it possible to use UUIDs, not just device names, and this is a nice improvement. However, in the process, this also means that the check that the symlink actually points to a device (not a partition) is now effective.
• It so happens that various users have started relying on that "let's rename the symlink to fool the driver" behaviour, and despite the solution used was not made for this, it worked to some extent.
• It still works with entire disks.

From here, how to go on?

1. If there are other suitable solutions to your needs, avoid the udev driver to attach whole disks to VMs.
2. If you really have to use the udev driver in this unsupported way, use entire disks, not partitions.
3. If you really have to use partitions, and I don't recommend it, you could likely cheat by creating a loopback device for the disk partition and use this as a device for the SR (also ensuring it's correctly setup at boot).
4. If we can understand the use case of associating disk partitions to VMs, maybe we'll find a way to implement something cleaner. Or will suggest a better solution.