[Feature Request]: Keeping more messages in node memory than just last one

[pedjas]

Platform

Cross-Platform

Description

Make node keep more messages than just last one. If not used with mobile phone connected, messaging is more or less useless when only the last message is available.

[b8b8]

The last 30 messages should be stored on the node until reconnection to a phone.

[korbinianbauer]

What's the rationale behind 30? The nodes keep and send hundreds of NodeInfos to the companion App after all, why throw away the actual messages after just 30?

[Xaositek]

The line has to be drawn somewhere and 30 messages is around 15kb of RAM if memory serves me properly. Additionally there is coming support to read these messages directly on the screen.

#8182

[b8b8]

What's the rationale behind 30? The nodes send hundreds of NodeInfos to the companion App after all, why throw away the actual messages after just 30?

Notice you said APP.
The NODE can only store 80-250 NodeInfos depending on hardware and 30 messages.
Obviously the APP can store an unlimited amount of NodeInfos and messages.
We/I am only talking about the NODE HARDWARE as it has limited resources, and like said above, the line needed to be drawn somewhere. Devices are designed to attached to a phone most of the time.

[korbinianbauer]

Notice you said APP.
[...]
We/I am only talking about the NODE HARDWARE as it has limited resources

Yes, I got that, so am I.

The NODE can only store 80-250 NodeInfos depending on hardware and 30 messages.

I don't think scaling the number of stored NodeInfos according to available hardware up to 250, but drawing a hard line at 30 messages is a sensible trade-off.

I believe focusing on keeping so many Nodes in the NodeDB is an artefact from back when the App didn't store any (before App version ~2.5.20). Now that the App can store an arbitrarily large NodeDB, that's not necessary anymore.

the line needed to be drawn somewhere

I don't think so. You don't have to have a fixed allocation. When memory is filling up, just delete the oldest entry, whether that's a NodeInfo or a text message.

Makes no sense to keep the NodeInfo of a node you haven't heard in 8 months on the node, but discard a message from 10 minutes ago.

IF there's a hard line, it should be the other way around: Limit the number of NodeInfos and keep as many messages as possible for the companion App to fetch later

It could exempt favorited nodes' NodeDB entry from discarding, just like the App does.

Devices are designed to attached to a phone most of the time.

Yes. But there's no need to make the experience worse than it needs to be when it's not attached for some period.

[pedjas]

I have T114 with forward version 2.6.11. It shows only the last message. If new arrives, previous is not accessible any more. There is no option to list other messages, all I can see is the last one.

Maybe that is changed in alpha firmware?

[Xaositek]

I have T114 with forward version 2.6.11. It shows only the last message. If new arrives, previous is not accessible any more. There is no option to list other messages, all I can see is the last one.

Maybe that is changed in alpha firmware?

Please see above with #8182

[b8b8]

@korbinianbauer how much memory is left over on NRF devices to store messages. NodeInfo also stores keys. By purging the nodeinfo packets more frequently you are forcing all other nodes to resend said nodeinfo more often on the mesh. Nodeinfo is a heavy packet that makes up a good chunk of bandwidth already while text messages take up very little traffic comparatively.

"boring" non-favorited nodes are already automatically purged.

Reducing the node database size from 80 to 40 would effectively double the amount of nodeinfo traffic (on a busy mesh) would it not? All other things being equal.

Are you expecting to leave node unattended at home and then when you get back from work download all the day's messages? The store and forward module on a private network or a computer at home logging the messages would probably be better. In what instance would you be separated from your day-to-day device long enough that a 30 message buffer would be insufficient?

It actually makes more sense to increase the node database where possible to further reduce nodeinfo exchange on capable hardware.

[korbinianbauer]

My rationale is this:

If the node is not connected to the App, ideally I want to not lose any information for as long as possible until it reconnects. This is achieved by discarding information based on age only, not type.

Reducing the node database size from 80 to 40 would effectively double the amount of nodeinfo traffic would it not?

No, I don't think so, since we don't discard randomly, but the oldest ones, which are most likely already synced to the App and it's less likely to receive anything from them. No need to send NodeInfo-packets even more often.

NodeInfo also stores keys

We only need those keys to send messages to the node, not to decrypt its messages. The App (which still retains the NodeInfo) could make sure to send the NodeInfo to the node before sending the message, to ensure the node has the key required.

Maybe it already does, I'm not sure.

how much memory is left over on NRF devices to store messages

I don't have exact numbers, but here's my guesstimation:

• A NodeDB entry takes 100-200 bytes in ram depending on what details are known, length of name etc. let's say 150 bytes on average.

• A message probably has 50-300 bytes depending on length, but most messages are short, so I'd say 100 bytes on average is generous.

Right now we store 250 NodeDB entries (-> 37.5 KB) and 32 messages (3.1 KB), so I assume ~40.6 KB are used in total currently, and we don't want to use more than that.

We can use that to store ~400 messages or ~275 NodeDB entries, or half of each, etc.

[b8b8]

If your node is disconnected from the phone and there are more than 80 nodes in your mesh (most cities) you WILL discard and redownload nodeinfo from the "new" nodes discovered. Regardless of the nodes in the app since the app is not connected.

What is your actual use case though? You never answered that. In what situation are you keeping your node away from your phone for a long period that 30 messages isn't enough?

Meshtastic is generally designed to be attached to am app, unless you are specifically building a standalone unit with keyboard etc.

[korbinianbauer]

redownload nodeinfo from the "new" nodes discovered

Good point. The firmware could keep a compact list of known (to the App) Node IDs without keeping the whole NodeInfo-details to avoid unnecessary requests. 250 NodeIDs would be just 1 KB instead of 37 KB.

What is your actual use case though? You never answered that

The use case is receiving messages without requiring your phone to stay within Bluetooth range of some place that has good Mesh-Reception.

In what situation are you keeping your node away from your phone for a long period that 30 messages isn't enough?

Whenever I want the node to be in an exposed place with good reception, but I want to move around more than reliable Bluetooth range. E.g. leave it at a window, and move around in the house freely.

I feel like you assume receiving 30 messages takes a long time, but sometimes that's just 5-10 minutes.

What's the problem that is solved by keeping 250 NodeDB entries on the node?

[b8b8]

Perfect candidate for the new client_base role (basically a personal repeater for just your own nodes) and keep your personal node on you.

Also esp32 devices work with wifi so you can roam the whole house without losing signal. My comments were more for NRF nodes that have limiting resources. Esp32 nodes, especially the ones that can take an SD card shouldn't have a message limit where technically feasible.

[jp-bennett]

You should be able to do a custom compile, that overrides the MAX_RX_TOPHONE and MAX_NUM_NODES values. I don't think we're planning to change the defaults at this point, but I'd be interested to hear back how the experiment goes.

[korbinianbauer]

@jp-bennett can you comment on whether my guesstimation on memory footprint for NodeDB and message ringbuffer is reasonable?

I can try a customized firmware, but I don't think I have the tools to monitor memory usage.

I'm also not sure how MUI handles this, I probably have to get some other companion nodes than the T-Decks I currently use.

client_base ... wifi

That's a perfectly fine solution if I am in my own home, yes. Everywhere else, I don't have a wifi and I don't like to bring two nodes.