POSIX semantics and ZFS on synchronous + asynchronous writes

[dgiantsidi]

Hello,

ZFS (ZIL) does not guarantee any ordering between asynchronous and synchronous itxs even for the same object.

Example to clarify my understanding:
In a single-threaded application, we execute the following:

fd = ​open(file.txt)
write(fd, A1);
write(fd, A2);
close(fd);

fd_sync = open(file.txt | O_SYNC);
write(fd_sync, A3);
close(fd_sync);

fsync(fd_sync);

By reading the ZIL code it seems that write(fd, A1) and write(fd, A2) will be placed to the asynchronous itxs (avl_tree_t) and there respective ordering will be preserved (e.g., if write(fd, A2) is visible then write(fd, A1) has also took effect).

When we do the fsync we serialize the synchronous and asynchronous writes into the sync list; we append the asynchronous itxs into the synchronous itxs (at the end). Now, the third write(fd_sync, A3) will be placed to the synchronous itxs list. Basically I understand that the list with the synchronous itxs will now have write(fd,A3 -> write(fd, A1) -> write(fd, A2) in this particular order.

Is my understanding correct? Shouldn't a read always see the latest write? In case of a crash the latest write appears to be write(fd,A2).
What am I missing? :)

Thanks again,
Dimitra

[amotin]

The answer is inside zfs_open() implementations:

<------>/*                                                                                                                                                     
<------> * Keep a count of the synchronous opens in the znode.  On first                                                                                       
<------> * synchronous open we must convert all previous async transactions                                                                                    
<------> * into sync to keep correct ordering.                                                                                                                 
<------> */                                                                                                                                                    
<------>if (flag & O_SYNC) {                                                                                                                                   
<------><------>if (atomic_inc_32_nv(&zp->z_sync_cnt) == 1)                                                                                                    
<------><------><------>zil_async_to_sync(zfsvfs->z_log, zp->z_id);                                                                                            
<------>}                                                                                                                                                      

The moment you open file with O_SYNC, all the previous async writes turn sync, and any new writes should follow after.

[dgiantsidi]

Hi @amotin ,

Okay, thank you! What you are saying makes sense!

I am working on a previous version (tag: 2.2.4) which has slightly a different implementation for zfs_open() and zfs_write() that was why I was confused.

For example, zfs_open increases the counter (so z_sync_cnt != 0, when we open the file with O_SYNC)

zfs_open():
  /* Keep a count of the synchronous opens in the znode */
  if (flag & O_SYNC)
      atomic_inc_32(&zp->z_sync_cnt);

But when we write (zfs_write -> zfs_log_write) all itx are by default sync (itx_sync = B_TRUE), unless we enter this piece of code.
So I do not see how we serialize the writes correctly here.

zfs_log_write():
  if (!(ioflag & (O_SYNC | O_DSYNC)) && (zp->z_sync_cnt == 0) && (fsync_cnt == 0))
      itx->itx_sync = B_FALSE;

However, yes, in the current version, it does make sense.