file: Query physical block size and minimum I/O size

[tchaikov]

Enhance block device initialization to query additional device characteristics beyond logical block size:

• use physical_block_size for write alignment
• use logical_block_size for read alignment

[tchaikov]

This table shows the parameters resulting from various device types being registered with the Linux kernel.

Table 9: Common device types and their resulting parameters

Device	logical	physical	min_io	opt_io	align_off
Disk 512/512	512	512	512	0	0
Disk 512/4KiB	512	4096	4096	0	0
Disk 512/4KiB, 1-aligned	512	4096	4096	0	3585
Disk 4 KiB/4 KiB	4096	4096	4096	0	0
RAID0, 64 KiB × 4 drives	512	512	65536	262144	0
RAID1, 16 KiB	512	512	16384	0	0
RAID5, 8 KiB × 3 drives, 1-aln.	512	512	8192	16384	3584

quoted from https://people.redhat.com/msnitzer/docs/linux-advanced-storage-6.1.pdf, section 1.5

[avikivity]

physical_block_size should be the write alignment, but not the read alignment. There's no downside to reading a 512 byte logical sector from a 4096 byte physical sector disk.

Wrt write alignment, even there it's iffy. Writing 4096 avoids RMW but can generate space amplification. With the current exposed parameters, physical_block_size for writes is the best match. We may want to expose another write alignment (choosing a name will be hard) to indicate a non-optimal block write block size that is smaller than the other one.

[tchaikov]

Thanks for pointing out the read/write differentiation! I've updated the implementation:

• Read alignment: Now uses logical_block_size only (as you suggested - no downside to reading 512-byte sectors)
• Write alignment: Now uses physical_block_size (not max(logical, physical, min_io))

You're right about the space amplification issue. I verified this in the kernel source - the Linux kernel only enforces logical_block_size alignment for O_DIRECT (see block/fops.c:blkdev_dio_invalid()):

  return (iocb->ki_pos | iov_iter_count(iter)) &
          (bdev_logical_block_size(bdev) - 1);

This confirms that physical_block_size and min_io are optimization hints, not requirements. Using physical_block_size provides the best balance:

• Avoids hardware-level RMW (4K physical sectors)
• Prevents space amplification from RAID stripe alignment (min_io can be 64 KiB+)

For context, I found that RAID devices set min_io to the chunk/stripe size (see drivers/md/raid0.c:386, raid5.c:7748, raid10.c:4003), which would cause massive space waste if used for write alignment.

Regarding exposing another write alignment: This is an interesting idea. We could expose something like:

• disk_write_dma_alignment = physical_block_size (current, safe default)
• disk_write_dma_alignment_optimal = max(physical_block_size, min_io) (for apps willing to trade space for throughput)

However, I'm inclined to defer this until we have a concrete use case. Most Seastar applications probably want "do the right thing" rather than having to choose between alignment strategies.

[avikivity]

This table shows the parameters resulting from various device types being registered with the Linux kernel.

Table 9: Common device types and their resulting parameters

Device logical physical min_io opt_io align_off
Disk 512/512 512 512 512 0 0
Disk 512/4KiB 512 4096 4096 0 0
Disk 512/4KiB, 1-aligned 512 4096 4096 0 3585
Disk 4 KiB/4 KiB 4096 4096 4096 0 0
RAID0, 64 KiB × 4 drives 512 512 65536 262144 0

I don't understand min_io and opt_io for RAID0. The disks could just as easily read and write 512 byte blocks here.

RAID1, 16 KiB 512 512 16384 0 0

Nor this. Why is 16k more optimal than anything else?

RAID5, 8 KiB × 3 drives, 1-aln. 512 512 8192 16384 3584

16k makes sense here because it avoids a RMW. But I don't understand 8k.

quoted from https://people.redhat.com/msnitzer/docs/linux-advanced-storage-6.1.pdf, section 1.5

[tchaikov]

This table shows the parameters resulting from various device types being registered with the Linux kernel.
Table 9: Common device types and their resulting parameters
Device logical physical min_io opt_io align_off
Disk 512/512 512 512 512 0 0
Disk 512/4KiB 512 4096 4096 0 0
Disk 512/4KiB, 1-aligned 512 4096 4096 0 3585
Disk 4 KiB/4 KiB 4096 4096 4096 0 0
RAID0, 64 KiB × 4 drives 512 512 65536 262144 0

I don't understand min_io and opt_io for RAID0. The disks could just as easily read and write 512 byte blocks here.

RAID1, 16 KiB 512 512 16384 0 0

Nor this. Why is 16k more optimal than anything else?

RAID5, 8 KiB × 3 drives, 1-aln. 512 512 8192 16384 3584

16k makes sense here because it avoids a RMW. But I don't understand 8k.

quoted from https://people.redhat.com/msnitzer/docs/linux-advanced-storage-6.1.pdf, section 1.5

I looked into the Linux kernel source code to understand how these values are set. Here's what I found:

TL;DR: For RAID devices, min_io is set to the chunk/stripe unit size (the amount written to each individual disk), not the minimum physically possible I/O size.

In the Linux kernel, all RAID types set io_min to the chunk size:

• RAID0 (https://github.com/torvalds/linux/blob/master/drivers/md/raid0.c#L386):
  lim.io_min = mddev->chunk_sectors << 9;
  lim.io_opt = lim.io_min * mddev->raid_disks;

• RAID5 (https://github.com/torvalds/linux/blob/master/drivers/md/raid5.c#L7748):
  lim.io_min = mddev->chunk_sectors << 9;
  lim.io_opt = lim.io_min * (conf->raid_disks - conf->max_degraded);

• RAID10 (https://github.com/torvalds/linux/blob/master/drivers/md/raid10.c#L4003):
  lim.io_min = mddev->chunk_sectors << 9;
  lim.io_opt = lim.io_min * raid10_nr_stripes(conf);

Answering Your Questions

RAID0 (64 KiB × 4 drives): "The disks could just as easily read and write 512 byte blocks here."

Yes, physically they can, but:

• min_io = 65536 (64 KiB) is the chunk size - aligning I/O to chunk boundaries avoids fragmenting operations across multiple drives unnecessarily
• opt_io = 262144 (256 KiB) is the full stripe width (64 KiB × 4) - this maximizes parallelism by utilizing all drives

You're right that smaller I/Os work, but they defeat the purpose of RAID0's striping.

RAID1 (16 KiB): "Why is 16k more optimal than anything else?"

I couldn't find this in the kernel code. RAID1 doesn't set io_min to a chunk size (it has no striping concept). This 16k value is likely either:

• Inherited from the underlying disk's characteristics
• Controller/configuration specific
• Or possibly an artifact of how that specific example was set up

RAID1 should typically just mirror the underlying device topology.

RAID5 (8 KiB × 3 drives): "I don't understand 8k"

From the kernel code:

• min_io = 8192 (8 KiB) is the chunk size - smallest alignment unit
• opt_io = 16384 (16 KiB) is the data stripe width (8 KiB × 2 data drives, excluding parity)

The 8k ensures I/O aligns with chunk boundaries. The 16k (which you understood) is optimal because it writes a full data stripe.

[tchaikov]

v2:

• use physical_block_size for write alignment
• use logical_block_size for read alignment

[mykaul]

quoted from https://people.redhat.com/msnitzer/docs/linux-advanced-storage-6.1.pdf, section 1.5

This is a great document, but it's also ~15 years old. It's less relevant for SSD/NVMe, for example.

[tchaikov]

This is a great document, but it's also ~15 years old. It's less relevant for SSD/NVMe, for example.

Good point. While the document is dated, the kernel topology API it describes is still current and works for modern devices.

For modern SSDs/NVMe (verified from Linux kernel source drivers/nvme/host/core.c):

/* NOWS = Namespace Optimal Write Size */
if (id->nows)
    io_opt = bs * (1 + le16_to_cpu(id->nows));

• logical_block_size and physical_block_size: Usually 512 or 4096 bytes
• minimum_io_size: Set to physical_block_size (typically 512 or 4096)
• optimal_io_size: Derived from NOWS (Namespace Optimal Write Size) field in NVMe spec
  • If NOWS = 0 (not specified): optimal_io_size = 0
  • If NOWS is set (e.g., 7): optimal_io_size = block_size * (1 + NOWS) = 512 * 8 = 4096
  • Many consumer NVMe drives don't set NOWS, so optimal_io_size = 0
  • Enterprise/datacenter NVMe drives often set NOWS to indicate write granularity

The key difference from the RAID examples is that RAID arrays have complex stripe/chunk geometry (64 KiB chunks × multiple drives) leading to much larger min_io/opt_io values. NVMe drives, when they set NOWS, typically indicate page-size granularity (4-16 KiB).

Important: Our implementation in this PR deliberately uses only physical_block_size for write alignment, not optimal_io_size or minimum_io_size. This avoids write amplification that would occur with large optimal_io_size values (e.g., forcing all writes to 256 KiB alignment for RAID0 would be wasteful). We only need to avoid hardware-level RMW, which physical_block_size handles.

So while the document's RAID examples show legacy use cases with their stripe-based optimal sizes, we use the simpler physical_block_size approach that works well for both modern SSDs and RAID without write amplification issues.

[avikivity]

v2:

• use physical_block_size for write alignment
• use logical_block_size for read alignment

We recently discovered that some disks lie about physical_block_size.

@robertbindar (or @tchaikov if you want), suggest adjusting iotune to detect the physical block size and write it in io_properties.yaml. Then the reactor can pick it up and use it to override what it detects from the disk.

[avikivity]

memory_dma_alignment is fixed 512 IIRC regardless of logical/physical sector sizes.

[avikivity]

I see it can be obtained via statx stx_dio_mem_align.

[avikivity]

Interestingly it returns 4 for my nvme (and 512 for xfs files)

[tchaikov]

Interestingly it returns 4 for my nvme (and 512 for xfs files)

stx_dio_mem_align is always 4 for NVMe devices.

1. NVMe driver rets dma_alignment

file: /home/kefu/dev/linux/drivers/nvme/host/core.c

static void nvme_set_ctrl_limits(struct nvme_ctrl *ctrl,
              struct queue_limits *lim)
{
      lim->max_hw_sectors = ctrl->max_hw_sectors;
      lim->max_segments = min_t(u32, USHRT_MAX,
              min_not_zero(nvme_max_drv_segments(ctrl), ctrl->max_segments));
      lim->max_integrity_segments = ctrl->max_integrity_segments;
      lim->virt_boundary_mask = NVME_CTRL_PAGE_SIZE - 1;
      lim->max_segment_size = UINT_MAX;
      lim->dma_alignment = 3;  // <-- LINE 2080
}

2. block layer reads it and sets stx_dio_mem_align

File: /home/kefu/dev/linux/block/bdev.c

void bdev_statx(const struct path *path, struct kstat *stat, u32 request_mask)
{
      ...
      if (request_mask & STATX_DIOALIGN) {
              stat->dio_mem_align = bdev_dma_alignment(bdev) + 1;  // 3 + 1 = 4
              stat->dio_offset_align = bdev_logical_block_size(bdev);
              stat->result_mask |= STATX_DIOALIGN;
      }
      ...
}

so, it's indeed 4 bytes. will prepare a new revision of this pull request to use statx() to query memory buffer alignment for block devices, and fall back to physical_block_size if statx unavailable or unsupported

[tchaikov]

v2:

• use physical_block_size for write alignment
• use logical_block_size for read alignment

We recently discovered that some disks lie about physical_block_size.

@robertbindar (or @tchaikov if you want), suggest adjusting iotune to detect the physical block size and write it in io_properties.yaml. Then the reactor can pick it up and use it to override what it detects from the disk.

will see what i can do. the io performance is critical to crimson project. so this is important to us.