Message order mixup

[Wurzelkoch]

I just sent a message on delta desktop while the conversation was scrolled to an earlier date. The now sent message got sorted to the end of the window as it was scrolled, between messages I received more than a week ago instead of the end of the conversation. I stays there even across closing and reopening the app.

Affected version:

Delta Chat Desktop 1.46.1 (git: v1.33.0-881-g899f196d)
Delta Chat Core v1.140.2
SQLite 3.39.4
electron 30.0.6
node 20.11.1

[iequidoo]

I don't think it's a Desktop issue and caused by scrolling up the chat. Could you also provide "Info" for the last two messages? It's in the context menu. Particularly, there are "sent" and "received" timestamps and also "messageId" which may be interesting.

Also please note that the current Delta Chat Desktop version is 1.46.8 already.

[Wurzelkoch]

The mis-sorted message's info is:

Sent: 2024.09.13 19:34:53
State: Delivered, Encrypted

Message-ID: Mr.lFkw_qZArDD.IUfO5u877Jq@localhost
</INBOX/;UID=1347>

messageId 2251

It got sorted between

Sent: 2024.08.17 14:13:30 
Received: 2024.08.17 14:13:35
State: Seen, Encrypted

Message-ID: Mr.e8FSkLMPX9-.3PnBl7q-f9R@localhost
</INBOX/;UID=1282>

DKIM Results: Passed=true

messageId	2247

and

Sent: 2024.09.04 08:45:35 
Received: 2024.09.13 19:35:23
State: Seen, Encrypted

Message-ID: Mr.IrE2MZrxvjG.Jjnee1PzC7B@localhost
</INBOX/;UID=1317>

DKIM Results: Passed=true

messageId	2288

Right after the mis-sorted message I sent another one:

Sent: 2024.09.13 19:36:45
State: Delivered, Encrypted

File: /home/user/.config/DeltaChat/accounts/73f37ef7-f1a3-4ca2-a5ae-b556bc07af8f/dc.db-blobs/paste.png, 225767 bytes
Type: Image
Mimetype: image/png
Dimension: 1010 x 768

Message-ID: Mr.MCNQj148BD8.Wsypn1xPzRF@localhost
</INBOX/;UID=1348>

messageId	2324

and got a reply:

Sent: 2024.09.13 19:36:54
Received: 2024.09.13 19:36:59
State: Seen, Encrypted

Message-ID: Mr.rIRBzoFZyhd.vox-pcHgbpV@localhost
</INBOX/;UID=1349>

DKIM Results: Passed=true

messageId	2325

[Wurzelkoch]

Oh and the last message I sent before the mis-sorted one was:

Sent: 2024.09.04 12:45:33 
State: Delivered, Encrypted
Reactions: ❤️1

Message-ID: Mr.lOeyrXykRAY.aTt7dTfE-EA@localhost
</INBOX/;UID=1341>

DKIM Results: Passed=true

delivered Friday, 2024-09-13 19:35 (1726248924000)
messageId	2312

[iequidoo]

The reason is that </INBOX/;UID=1317> was received after </INBOX/;UID=1347> was sent (the former has greater messageId also confirming that). But Delta Chat displays "sent" timestamps in the chats as they seem more informative. That's why the messages are reordered. This is a known problem and i don't have idea how to improve it -- if we sort "old" received messages before locally sent ones, the user may not notice them, so it's better to "reorder" and make them more visible.

[iequidoo]

Oh and the last message I sent before the mis-sorted one was:

Sent: 2024.09.04 12:45:33 
State: Delivered, Encrypted
Reactions: ❤️1

Message-ID: Mr.lOeyrXykRAY.aTt7dTfE-EA@localhost
</INBOX/;UID=1341>

DKIM Results: Passed=true

delivered Friday, 2024-09-13 19:35 (1726248924000)
messageId	2312

This is strange because it has messageId > the one for the mis-sorted message (2251). Did you do any backup restorations (asking just in case, they shouldn't cause such effects)?

[Wurzelkoch]

Hm, looking at the details you spotted, I do have a theory:
It's possible that I sent or received messages 2288 to 2312 on the mobile app (android), then opened the desktop app, sent 2251 and only then did the desktop app receive 2288 to 2312 and I just confabulated that the conversation window was scrolled to a random place. Because I certainly remember reading and answering to 2288 on Sep 4th, and other than switching between parallel devices I did no backup restoration or the like. On the mobile app they are also in the right order.

[iequidoo]

It's possible to check which messages were sent from another device -- they have a line containing "received", you should look at the bottom of the "Message Details" window. Smth like
received Thursday, September 5, 2024 8:19 PM (1725578348000)
Particularly, </INBOX/;UID=1347> and </INBOX/;UID=1317> messages are interesting ones.

[iequidoo]

It's possible that I sent or received messages 2288 to 2312 on the mobile app (android), then opened the desktop app, sent 2251 and only then did the desktop app receive 2288 to 2312

Yes, this may be the reason, but even then i don't have a solution for the problem because well, </INBOX/;UID=1317> was seen on another device, but not not on this one, so it should be sorted to the bottom because it's assumed that the account may be shared and devices are used by different people. Though this is not a common use of Delta Chat, but still. Or devices may be used by the same person, but in the different mind states :/

EDIT: I think that by default Delta Chat should assume that account isn't shared, then received outgoing messages and seen incoming ones may be sorted as non fresh and appear somewhere in the middle of the chat. If an account is shared, that should be set by some advanced setting in the UI i think. If we go for this or some other solution, this issue may be closed.

[link2xt]

While we can change how sorting of seen incoming messages works, it is not clear that it is an improvement. I think it's better not to complicate the logic by depending on the \Seen flag as it will make debugging ordering issues and figuring out what happened more difficult as now you need to know which state the flag was in. This may get even more complicated with multitransport and servers having different states.

[iequidoo]

In fact we already complicate sorting with the \Seen flag because we don't check it for a new message arrived, but we add it in a InSeen state and this affects sorting of messages arriving after that message. But i agree that we shouldn't complicate it even more, it's not clear if we want to depend on the \Seen flag. As said above, the user can share their account with someone or a bot. Btw, in case of a bot we shouldn't mark messages as seen at all, it seems to have no value. Sending MDNs is enough.
...
Made a PR for this: #7195