Handling non-blocking file descriptors consistently.

[ioquatix]

This is more of a discussion than an actual issue at the moment, but I hope to provide more concrete details as we explore the use cases.

We have an interesting problem regarding IORING_OP_READV/IORING_OP_WRITEV. In a general programming language environment, we don't always know if the file descriptor is in non-blocking mode or not.

But the behaviour of IORING_OP_READV/IORING_OP_WRITEV depends on it. In non-blocking mode, we can get a result of EAGAIN.

However, this is not very useful in the case of even driven concurrency where we want operations like this to be "asynchronous" "blocking" - i.e. the CQE should not be generated until the read/write could make progress.

In general, my understanding is, we'd need 3-4 system calls to work around this, something like this:

int current_state = get_nonblocking_state(fd);
if (current_state != nonblocking) set_nonblocking_state(fd);
io_uring_read(..., fd, ...)
if (nonblocking_state_was_changed) restore_nonblocking_state(fd, current_state);

We can cache this to reduce the round trips, but it feels like this would be something better solved by a flag on IORING_OP_READV/IORING_OP_WRITEV. In fact, I'm having a hard time imagining any valid use case for non-blocking operation in the uring.

I'll try to test out specific cases where this is a problem and report back.

[YoSTEALTH]

Whole point of io_uring is its asynchronous by design! If there is a blocking call it either handles in async or opens a new kernel thread for that request.

By using O_NONBLOCK what you are saying is, let me choose what to do next when blocking happens vs letting io_uring deal with it.

There already is an flags option.

fd = open('path/to/file', O_RDONLY)  # normal blocking read only open.
...
io_uring_prep_readv(sqe, fd, iovecs, nr_vecs, offset)
sqe.rw_flags |= RWF_NOWAIT  # now read is non-blocking
...
# catch `-EGAIN` in `cqe.res`, ...

[ioquatix]

Is there a flag for the opposite of RWF_NOWAIT?

[YoSTEALTH]

Not that I know of. If you want a blocking read just use normal read() don't use io_uring_read*

[ioquatix]

We have file descriptors that might already have O_NONBLOCK set. But we still want to use them with io_uring without getting EAGAIN.

[YoSTEALTH]

Hmm that's a tricky situation. Monkey patching O_NONBLOCK flag to not be non-blocking might lead to other problems/bugs.

If you have access to when the original flag is being set

original_flags = O_CREAT | O_RDWR | O_NONBLOCK

You could add a check like

if original_flags & O_NONBLOCK:
    # raise error saying `O_NONBLOCK` is not supported, invalid or something!

I am not really proud of that solution.

Why not just catch -EGAIN and poll it? io_uring_prep_poll_add(sqe, fd, POLLIN)

[ioquatix]

Yes it’s one option but we wanted to use the direct read/write operations if possible.

[YoSTEALTH]

suppose you can also remove the flag by:

original_flags = O_CREAT | O_RDWR | O_NONBLOCK
new_flags = original_flags & ~O_NONBLOCK

[axboe]

It would be a pretty trivial change to arm poll rather than return -EAGAIN to the application, if O_NONBLOCK is set on the file. The main issue here is how to inform io_uring of that desired behavior. OP_READV/OP_WRITEV flags are really readv2/writev2 flags, so it'd then be a generic change to allow this. Which arguably would make sense, as the justification would be that you can do blocking IO without having the query/set new file flags.

This applies to more than OP_READV/OP_WRITEV, though. It also applies to SEND/RECV and friends, so it's actually more of a generic change. Which means that an IOSQE_ flag would make more sense. The only potential issue there is that the flag space is pretty small, we have 8 bits and 6 are already used. Not a huge issue, we have two bits left, and the last bit could always be turned into a "look at extended bits" or something like that.

In any case, let me know if you want to test a patch for this. Should be easy to cook up.

[ioquatix]

I am happy to test and help with this.

[ioquatix]

Just thinking through what you described, do you think it would make sense for us to propose a flag to readv2/writev2 to be more specific about this?

That way, this doesn't become an io_uring specific feature, but works at the kernel level in a generic way.

e.g.

       RWF_NONBLOCK
              Perform the operation as if the descriptor was in `O_NONBLOCK` mode.
       RWF_BLOCK
              Perform the operation as if the descriptor was not in `O_NONBLOCK` mode.

Maybe it's crazy proposal, but that seems logical to me, at least from my POV.

[axboe]

As mentioned, that only then works for the read/write variants, not for any other opcode. Hence I'd be leaning towards making it an sqe flag instead. If it was just applicable to OP_READV and friends, then the RWF approach would be preferable.

[ioquatix]

Do other opcodes also take flags? e.g. accept, open, close, etc.

[axboe]

Each opcode has a specific set of flags, like the RWF_* for read/write, but there's also a generic set of IOSQE_* flags that apply to any opcode. That handles things like links, etc. Things that sit above the specific request type.

[ioquatix]

Oh that makes total sense. Okay, I'm keen to test this when you are ready.

cc @ciconia

[axboe]

I’ll spin a patch tomorrow morning.

…

On Jun 16, 2021, at 5:56 PM, Samuel Williams ***@***.***> wrote:  Oh that makes total sense. Okay, I'm keen to test this when you are ready. cc @ciconia — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.