Should tcp.ipv4.tcp.rmem be set higher than net.core.rmem_max?

[Forza-tng]

Hi.

I notice that bpftune keeps increasing net.ipv4.tcp.rmem, even though it is beyond net.core.rmem_max. Is this correct behaviour?

Jul 22 14:43:43 e350 bpftune[55954]: bpftune works fully
Jul 22 14:43:43 e350 bpftune[55954]: bpftune supports per-netns policy (via netns cookie)
Jul 22 14:43:44 e350 bpftune[55954]: Scenario 'specify TCP congestion control algorithm' occurred for tunable 'net.ipv4.tcp_allowed_congestion_control' in global ns. To optimize TCP performance, a TCP congestion control algorithm was chosen to mimimize round-trip time and maximize delivery rate.
Jul 22 14:43:44 e350 bpftune[55954]: updating 'net.ipv4.tcp_allowed_congestion_control' to 'reno bbr cubic dctcp htcp'
Jul 22 14:44:00 e350 bpftune[55954]: Scenario 'need to increase TCP buffer size(s)' occurred for tunable 'net.ipv4.tcp_rmem' in global ns. Need to increase buffer size(s) to maximize throughput
Jul 22 14:44:00 e350 bpftune[55954]: Due to need to increase max buffer size to maximize throughput change net.ipv4.tcp_rmem(min default max) from (4096 131072 91552733) -> (4096 131072 114440916)
Jul 22 14:44:30 e350 bpftune[55954]: Scenario 'need to increase TCP buffer size(s)' occurred for tunable 'net.ipv4.tcp_rmem' in global ns. Need to increase buffer size(s) to maximize throughput
Jul 22 14:44:30 e350 bpftune[55954]: Due to need to increase max buffer size to maximize throughput change net.ipv4.tcp_rmem(min default max) from (4096 131072 114440916) -> (4096 131072 143051145)
Jul 22 14:44:30 e350 bpftune[55954]: Scenario 'need to increase TCP buffer size(s)' occurred for tunable 'net.ipv4.tcp_rmem' in global ns. Need to increase buffer size(s) to maximize throughput
Jul 22 14:44:30 e350 bpftune[55954]: Due to need to increase max buffer size to maximize throughput change net.ipv4.tcp_rmem(min default max) from (4096 131072 143051145) -> (4096 131072 178813931)
Jul 22 14:44:30 e350 bpftune[55954]: Scenario 'need to increase TCP buffer size(s)' occurred for tunable 'net.ipv4.tcp_rmem' in global ns. Need to increase buffer size(s) to maximize throughput
Jul 22 14:44:30 e350 bpftune[55954]: Due to need to increase max buffer size to maximize throughput change net.ipv4.tcp_rmem(min default max) from (4096 131072 178813931) -> (4096 131072 223517413)
Jul 22 14:44:30 e350 bpftune[55954]: Scenario 'need to increase TCP buffer size(s)' occurred for tunable 'net.ipv4.tcp_rmem' in global ns. Need to increase buffer size(s) to maximize throughput
Jul 22 14:44:30 e350 bpftune[55954]: Due to need to increase max buffer size to maximize throughput change net.ipv4.tcp_rmem(min default max) from (4096 131072 223517413) -> (4096 131072 279396766)
Jul 22 14:56:43 e350 bpftune[55954]: Scenario 'need to increase TCP buffer size(s)' occurred for tunable 'net.ipv4.tcp_rmem' in global ns. Need to increase buffer size(s) to maximize throughput
Jul 22 14:56:43 e350 bpftune[55954]: Due to need to increase max buffer size to maximize throughput change net.ipv4.tcp_rmem(min default max) from (4096 131072 279396766) -> (4096 131072 349245957)

My system has the following max set.

❯ sysctl net.core.rmem_max
net.core.rmem_max = 7500000

According to https://www.kernel.org/doc/html/latest/networking/ip-sysctl.html tcp.rmem is capped at core.rmem_max and ditto for wmem.

[Forza-tng]

I am thinking that bpftune probably should look at actual socket memory buffers in-use, and not the autotuned maximum buffer size. I.e. even if we have a certain maximum, the apps might still read packets fast enough so the buffer never gets near full. So, the kernel might have set a high maximum -> triggering bpftune to increase tcp_rmem, buf the buffers aren't actually used.

It seems unlikely that any app was unable to de-queue packets fast enough on my system to warrant gigabytes large buffers. Perhaps, instead of reading only the allocations, bptune could read actual buffer usage for each socket and weigh that in before increasing tcp_rmem and tcp_wmem.

Jul 27 10:55:50 e350 bpftune[81568]: no tuner netns found for tuner 8, cookie 1
Jul 27 10:55:50 e350 bpftune[81568]: covar for 'net.ipv4.tcp_rmem' netns 1 (new 4096 131072 1009741238): -4292629599790.382883 ; corr -0.560565
Jul 27 10:55:50 e350 bpftune[81568]: no tuner netns found for tuner 8, cookie 1
Jul 27 10:55:50 e350 bpftune[81568]: no tuner netns found for tuner 8, cookie 1
Jul 27 10:55:50 e350 bpftune[81568]: writing sysctl '/proc/sys/net/ipv4/tcp_rmem' for netns_fd 0
Jul 27 10:55:50 e350 bpftune[81568]: set caps (count 1)
Jul 27 10:55:50 e350 bpftune[81568]: set caps (count 2)
Jul 27 10:55:50 e350 bpftune[81568]: Read net.ipv4.tcp_rmem[0] = 4096
Jul 27 10:55:50 e350 bpftune[81568]: Read net.ipv4.tcp_rmem[1] = 131072
Jul 27 10:55:50 e350 bpftune[81568]: Read net.ipv4.tcp_rmem[2] = 807792991
Jul 27 10:55:50 e350 bpftune[81568]: drop caps (count 1)
Jul 27 10:55:50 e350 bpftune[81568]: Wrote net.ipv4.tcp_rmem[0] = 4096
Jul 27 10:55:50 e350 bpftune[81568]: Wrote net.ipv4.tcp_rmem[1] = 131072
Jul 27 10:55:50 e350 bpftune[81568]: Wrote net.ipv4.tcp_rmem[2] = 1009741238
Jul 27 10:55:50 e350 bpftune[81568]: drop caps (count 0)
Jul 27 10:55:50 e350 bpftune[81568]: Scenario 'need to increase TCP buffer size(s)' occurred for tunable 'net.ipv4.tcp_rmem' in global ns. Need to increase buffer size(s) to maximize throughput
Jul 27 10:55:50 e350 bpftune[81568]: Due to need to increase max buffer size to maximize throughput change net.ipv4.tcp_rmem(min default max) from (4096 131072 807792991) -> (4096 131072 1009741238)
Jul 27 10:55:50 e350 bpftune[81568]:  updated stat for tunable net.ipv4.tcp_rmem, scenario 0: 37
Jul 27 10:55:50 e350 bpftune[81568]: event scenario [0] for tuner tcp_buffer[8] netns 1 (global netns) sent

[alan-maguire]

yeah, at the momemt the analysis is rather shallow; it looks at the worst offender for memory utilization and increases buffers based on that. now that's not great of course because it's not representative of the system as a whole. The big problem is that we're event-driven so at the time we handle the event for the socket we don't have context across other sockets. A more weighted approach would certainly be worth considering.

on the tcp rmem max being higher than the socket-general value a good description is here https://stackoverflow.com/questions/31546835/tcp-receiving-window-size-higher-than-net-core-rmem-max . Basically the tcp value wins as an upper limit.