usb: add USB mass storage class support

[mikesmitty]

I've only tested it at a very basic level so far, but it's a somewhat faithful port of TinyUSB's mass storage class support and it shows up as a USB disk without errors in linux. There are still a number of things to be cleaned up, a few bits ironed out (clearing stalls doesn't work yet, for example), and I'll soon be implementing a FAT12 emulation package to go with it, but I figured I'd push it up as a draft for any comments

This is a basic program that runs it:

package main

import (
	"machine/usb/msc"
	"time"
)

func main() {
	m := msc.Port()
	for {
		println(m.State())
		time.Sleep(2 * time.Second)
	}
}

Edit: Oh, and tests will be coming as well

[b0ch3nski]

@mikesmitty cool stuff!

Regarding FAT emulation, have you seen @soypat repo? https://github.com/soypat/fat :)

[deadprogram]

I would suggest looking at https://github.com/tinygo-org/tinyfs/tree/release/fatfs since it is already being supported in tinyfs, which would probably be an important user of this feature.

[mikesmitty]

@b0ch3nski @deadprogram Thanks for the info, I should have probably explained a bit more. My goal is to emulate FAT12 with a TinyFS backend. My personal desired use case is to use LittleFS on disk, but present an emulated FAT12 over USB

[mikesmitty]

As I'm thinking about it, adding the FAT12 emulation bits probably make more sense in the TinyFS repo itself. This is the interface I put together for interfacing with the USB SCSI commands. It seems fine to me at first glance, but since it'd be a hard-to-change interface I want to make sure it makes sense:

type Disk interface {
	Ready() bool
	ReadOnly() bool
	BlockCount() uint32
	BlockSize() uint32
	Read(offset uint32, buffer []byte) (uint32, error)
	Write(offset uint32, buffer []byte) (uint32, error)
}

The big potential issue that I can foresee is that because I'm using a uint32 for the offset in Read/Write any emulated disks will be limited to a max of ~4GB in size. I originally was trying to allow it to use io.ReaderAt/io.WriterAt which would require converting the sector + offset used internally into a plain offset, but in hindsight I think that isn't a useful goal given the other required parts of the interface. Since it's already more than just throwing a simple reader/writer at it, abstracting away the concept of sectors and as a result limiting potential max disk size probably isn't very useful

[soypat]

I arrived at the following abstraction for a block device. I'd omit Ready and ReadOnly methods for now. Maybe there's a better way of doing things leveraging abstractions in other Go standard library packages such as the fs package for permissions.

type BlockReader interface {
	ReadBlocks(dst []byte, startBlock int64) (int, error)
}

type BlockWriter interface {
	WriteBlocks(data []byte, startBlock int64) (int, error)
}

// BlockDevice is the basic interface that must be implemented for a device to function properly.
type BlockDevice interface {
	BlockReader
	BlockWriter
	EraseBlocks(startBlock, numBlocks int64) error
	BlockSize() int
	BlockCount() int64
}

[mikesmitty]

Thanks for weighing in @soypat, I could roll with that interface and those suggestions. I think I'll need to implement the UNMAP SCSI command in order for EraseBlocks() to be usable, but given that just about everything this will write to is probably going to be flash storage that's probably for the best anyway. I realized afterwards that it's a pretty likely that this would at some point be used to provide access to an SD card over USB so artificially restricting the interface to low-GB volume sizes instead of TB volume sizes is definitely sub-optimal

[mikesmitty]

So I completely forgot that machine.BlockDevice exists while thinking I needed to create another blockdevice-type interface, and that the erase block size vs write block size is also super important for discard operations. I'm super flexible about whatever shape the interface ends up taking

[soypat]

I guess do with BlockDevice for now since its in the machine package

[deadprogram]

Once #4850 has had the merge conflicts resolved, and also merged, this PR will need the same.

It should, at that point, be a lot easier to review.

Is there some related code that you are using to test out the full file system mount etc @mikesmitty ?

Can I just say now very excited I am about this work, and how much I want/need to use it! 😻

[mikesmitty]

Is there some related code that you are using to test out the full file system mount etc @mikesmitty ?

I'm still working through my port (though it's turning into a bit of a rewrite as it goes on) of the FAT12 emulation layer in https://github.com/oyama/pico-littlefs-usb, but for testing in the mean time I've been using sg_vpd from the sg3-utils tool suite to verify the VPD communication fields:

• Supported VPD pages: sg_vpd -p sv /dev/sda
• Block Device Characteristics: sg_vpd -p bdc /dev/sda
• Block Limits: sg_vpd -p bl /dev/sda
• Logical Block Provsioning: sg_vpd -p lbpv /dev/sda
• Show all VPD page data: sg_vpd -a /dev/sda
  There's also a -H flag to provide a hex output of the returned data if that's useful for CI/CD later

And for testing the UNMAP/discard operations I used the blkdiscard tool, but found that I needed to manually enable UNMAP for the disk through sysfs first:

sudo sh -c 'echo "unmap" >/sys/block/sda/device/scsi_disk/0:0:0:0/provisioning_mode'

Because of the small size of the emulated disk I was only able to get blkdiscard to test single-pass, full-device discard operations so multi-packet UNMAP commands are as of yet untested (the else block on line 62 here: https://github.com/tinygo-org/tinygo/pull/4844/files#diff-28a9c9ce5c40a794d3a2b9f2db9aff7efae6a0f21d4420534cbc8a7243d41dfbR62), but single-packet commands worked fine. If I can't find a way of testing multi-packet UNMAP commands we can also just simplify it down to accept only single-packet UNMAP commands and set the max unmap descriptor block count here to 3: https://github.com/tinygo-org/tinygo/pull/4844/files#diff-7229572bc6cfb2b3f9b13656ea876d7612ec27a3c20e255e8c632bd9436b95aeR31

For read/write commands I've just been using dd to/from the emulated disk, though the emulated disk I'm using doesn't properly verify if writes are occurring, just that it's not throwing errors. Since it's using machine.BlockDevice now I'll have to put together some code to do some read/write/verification to the flash on a pico to make sure the writes are happening as intended

All that said, of course after giving it another read through I noticed a number of comments and little issues that need to be updated/fixed. Once I have a chance to look over the review in #4850 and that gets merged I'll rebase and fix those

[deadprogram]

Now that #4850 has been merged into dev this PR will need to be rebased @mikesmitty if you please.

Getting closer! 😅

[mikesmitty]

Doing some follow-up tests with a real backing flash device I'm seeing some errors. I'll need to do some diagnosing and cleanup before it's ready to be reviewed