io_uring_prep_sendmsg (readv/writev) atomicity guarantees

[gmelikov]

tl;dr What's the guarantee about io_uring_prep_sendmsg atomicity?

I try to make async network (TCP stream socket) protocol with "iodepth" (client and server are sequential and have only one thread, but they may generate more than 1 "async" request, you can think about it as a storage protocol, maybe like nvmeotcp), so there may be X commands incoming, and I wanted to answer to them via io_uring_prep_sendmsg, one by one, with some header and payload. In this case order is not critical, but atomicity is - each response header should have corresponding data with it and be whole.

I use

	struct msghdr msg = {
		.msg_iov = ioreq->iovecs, // My "header + data" is here in different iovecs
		.msg_iovlen = 3};

	struct io_uring_sqe *sqe = get_sqe(&ring); // Basic wrapper on io_uring_get_sqe
	io_uring_prep_sendmsg(sqe, fd, &msg, MSG_WAITALL | MSG_NOSIGNAL);
	io_uring_sqe_set_data(sqe, ioreq);
	io_uring_submit(&ring);

, where I send data in one uring SQE and submit it before any manipulations with buffer. BUT when I submitted more than 22 such inflight SQEs with 4+KB of data each, on 1500MTU physical network between client<->server I've got some racy mixed up data from several sendmsgs intersected. I see that CQEs come in different order and that's ok, but send data atomicity is a problem. Maybe it's a question about MSG_WAITALL.

I can't reproduce it if server and client are on the same machine even with 64 "parallel" inflight io_uring_prep_sendmsg SQEs.
Just wanted to check my sanity, thank you.

[isilence]

If the file provides no atomicity guarantees, there can be no atomicity with io_uring requests.

If that's about ordering the execution of io_uring requests, there has already been a couple of threads answering in more details, but in short, request can be executed in any order and they can even be executed in halfs. So, totally possible that one request pushes its first half of data, then another request gets executed, and only then the first requests completes. Userspace must order requests in userspace.

[gmelikov]

Thank you for confirmation!

FWIW, with TCP sockets while send/recv buffers are not full, on modern kernels (6.8+) I consistently have fully-performed SQEs and don't have partial executions. Interesting if there'll be other cases which would force a partial SQE completion. For now I'll try to set appropriate socket buffers and check for data validity via checksums on recipient. If these cases are rare enough - I can live with that.

tl;dr: with tcp sockets set appropriate SO_SNDBUF/SO_RCVBUF helped me to deal with it, will see what fails next (and it should explode after all sometimes), and added data integrity checks on application level via checksums.