Questions about zil+zio layer

[dgiantsidi]

Hello,

When computing the ZIL block checksum ( using zio_checksum_compute, specifically when checksum == ZIO_CHECKSUM_ZILOG2 which eventually calls abd_fletcher_4_native), does this process always occur in a single-threaded context? For example, if we need to flush three log write blocks (LWBs), resulting in the creation of three ZIO events with ordering constraints (due to parent/child dependencies enforced by the ZIL codebase, zil_lwb_write_issue), are these tasks executed sequentially or in a pipelined manner to maintain dependencies? OR, can the checksums of dependent block be computed in parallel, allowing, let's say, for the second block checksum to finish before the first?

I understand there are pipeline stalls to ensure that child nodes complete before their parents (e.g., VDEV I/O start/done/assess). However, it is unclear whether the checksum computation respects this ordering (has pipeline stalls).

The reason I am asking is that I am building a more robust ZIL-chain where each ZIL-block depends on the checksum of the previous block. I am implementing that using a structure for global state in the zio_checksum.c file. My implementation assumes that by the time we calculate the checksum of block i, checksums for block i-1 and all preceding blocks have already been computed. Is this assumption correct?

[dgiantsidi]

Hello, this is a friendly reminder :)
If someone could clarify that would be great

[snajpa]

ping @amotin (IMO he's the most qualified of those who are regularly around here currently)

[amotin]

Hi Dimitra,

IIRC ZIL code itself does not serialize its ZIOs, issuing all of them it has as soon as it can. But it heavily relies on general ZIO pipeline rules. For example, in case of indirect ZIL write LWB ZIO must not start its processing until all the data blocks it references are compressed, encrypted, checksumed and allocated, otherwise it won't have the block pointer to include into the LWB. This wait is implemented by zio_wait_for_children(ZIO_WAIT_READY) call inside zio_write_compress(), which is the first stage of ZIO write pipeline that require the data.

When we are talking about two consecutive LWBs, formally they should not depend on each-other for data, so it would be great if they were checksumed in parallel, not only issued to the disks. But I suspect they follow the same rules as any other children and parents, which means the following (parent) LWB ZIO will wait at ZIO_STAGE_WRITE_COMPRESS stage (preceding the ZIO_STAGE_CHECKSUM_GENERATE stage) until the previous (child) LWB ZIO reach its ZIO_STAGE_READY stage, following the ZIO_STAGE_CHECKSUM_GENERATE. So in that way I suspect the answer to your question is: their checksuming is serialized now.

But I am not happy about this observation, and rather than fixate it that way by introducing the mentioned chaining, I'd honestly prefer the dependency to be disabled somehow to parallelize the processing. Otherwise ZIL write throughput is limited to a checksuming throughput of single CPU, which may be a problem in some configurations. Previously before my work around 2.2 we were limited by memory copy throughput of single CPU (which is not brilliant sometimes), when LWB population was done under ZIL lock. I've fixed the locking back then, so we no longer have that lock contention. But looking on this now, I suspect this ZIO serialization might be the next throughput limitation of synchronous writes, unless I am wrong somehow in all the above.

[dgiantsidi]

Hi,

Thanks much for the reply -- some minor questions (also I am using the version with tag: zfs-2.2.4)

(1)
Regarding these:

• This wait is implemented by zio_wait_for_children(ZIO_WAIT_READY) call inside zio_write_compress(), which is the first stage of ZIO write pipeline that require the data.

• But I suspect they follow the same rules as any other children and parents, which means the following (parent) LWB ZIO will wait at ZIO_STAGE_WRITE_COMPRESS stage (preceding the ZIO_STAGE_CHECKSUM_GENERATE stage) until the previous (child) LWB ZIO reach its ZIO_STAGE_READY stage, following the ZIO_STAGE_CHECKSUM_GENERATE.

Can you clarify what is the difference between: ZIO_DDT_WRITE_PIPELINE and ZIO_WRITE_PIPELINE. It seems that for the ZIL ZIOs, the ZIO_WRITE_PIPELINE is used where the checksum stage (ZIO_WRITE_COMMON_STAGES) takes place before the ZIO_STAGE_WRITE_COMPRESS. So I am not sure if the zio_wait_for_children(ZIO_WAIT_READY) in the zio_write_compress() ensures that checksuming takes place in a serial fashion.

(2)
Regarding this:
.. when LWB population was done under ZIL lock. I've fixed the locking back then, so we no longer have that lock contention.

I clearly see in the ZIL codebase that when the zil_lwb_write_issue is invoked the zilog->zl_issuer_lock is always held. Is this the lock you are talking about? It seems to me that the lwb's zio's are constructed and issued in a single-threaded context. In addition to this, when creating a new lwb we call into zil_lwb_set_zio_dependency which enforces the parent-childen constraints (thus, ordering constrains).

Copying the explanatory comment from the code in the file:

// Build the zio dependency chain, which is used to preserve the ordering of
// lwb completions that is required by the semantics of the ZIL. Each new lwb
// zio becomes a parent of the previous lwb zio, such that the new lwb's zio
// cannot complete until the previous lwb's zio completes.
 
// This is required by the semantics of zil_commit(): the commit waiters
// attached to the lwbs will be woken in the lwb zio's completion callback,
// so this zio dependency graph ensures the waiters are woken in the correct
// order (the same order the lwbs were created).

Is my understanding here correct in your opinion?

[amotin]

Can you clarify what is the difference between: ZIO_DDT_WRITE_PIPELINE and ZIO_WRITE_PIPELINE.

As you could guess, the first is used for dedup writes. It is similar to normal write up to the checksum stage, but then may skip physical write, replacing it with DDT update.

It seems that for the ZIL ZIOs, the ZIO_WRITE_PIPELINE is used where the checksum stage (ZIO_WRITE_COMMON_STAGES) takes place before the ZIO_STAGE_WRITE_COMPRESS.

The order they are listed in the macros does not matter. The stages are always executed according to their order in enum zio_stage.