Latency or no latency?

[bzed]

While looking at the logs I found the following part:

Dec 10 20:26:07 foobar bpftune[1050862]: Scenario 'need to retain TCP buffer size due to latency' occurred for tunable 'net.ipv4.tcp_rmem' in global ns. Latency is starting to correlate with buffer size increases, so do not make buffer size increase to avoid this effect
Dec 10 20:26:07 foobar bpftune[1050862]: Due to correlation between buffer size increase and latency change net.ipv4.tcp_rmem(min default max) from (4096 87380 32768000) -> (4096 87380 32768000)
Dec 10 20:26:07 foobar bpftune[1050862]: Scenario 'need to retain TCP buffer size due to latency' occurred for tunable 'net.ipv4.tcp_rmem' in global ns. Latency is starting to correlate with buffer size increases, so do not make buffer size increase to avoid this effect
Dec 10 20:26:07 foobar bpftune[1050862]: Due to correlation between buffer size increase and latency change net.ipv4.tcp_rmem(min default max) from (4096 87380 32768000) -> (4096 87380 32768000)
Dec 10 20:26:07 foobar bpftune[1050862]: Scenario 'need to increase TCP buffer size(s)' occurred for tunable 'net.ipv4.tcp_rmem' in global ns. Need to increase buffer size(s) to maximize throughput
Dec 10 20:26:07 foobar bpftune[1050862]: Due to need to increase max buffer size to maximize throughput change net.ipv4.tcp_rmem(min default max) from (4096 87380 32768000) -> (4096 87380 40960000)
Dec 10 22:01:34 foobar bpftune[1050862]: Scenario 'need to increase TCP buffer size(s)' occurred for tunable 'net.ipv4.tcp_rmem' in global ns. Need to increase buffer size(s) to maximize throughput
Dec 10 22:01:34 foobar bpftune[1050862]: Due to need to increase max buffer size to maximize throughput change net.ipv4.tcp_rmem(min default max) from (4096 87380 40960000) -> (4096 87380 51200000)

So first it was decided that larger buffers affect latency - and later it doesn't matter anymore?

[alan-maguire]

thanks for filing this. there's definitely a need to think more deeply about how we handle tuning down. when I last experimented with it the problem was that a correlation on the way up would persist on the way down and we would often overcorrect (i.e. tune up a bit then tune down a lot). A temporary workaround was to reset the correlation but this report illustrates the problem with that - we just start tuning up again. The challenge here is finding reproducible ways to test such correlations - we have tests for most tuned values, but most tests just set a value way too low and ensure it gets bumped up; it's harder to reliably reproduce correlations between tuned values and latencies which makes testing this harder. It may be best to revert to the "tune down while correlation between tunable changes and latency exists" approach, but ideally we'd need some tests to cover this behaviour.

[alan-maguire]

one thought is we develop models alongside tuners that attempt to simulate tuner behaviour. the problem of course is we can see divergence between such models and reality, especially when they dynamics are complex.

[alan-maguire]

for now I've restored the original behaviour via #134 ; so we tune down when we observe correlation between buffer size increases and latency until said correlation goes away. there's no easy way to force testing of this scenario I can think of yet but will see if I can come up with something..