Feature Request: tuner for netdev_max_backlog

[dkratunov]

This is a recent experience in a large k8s cluster receiving about a few gigabit per node with large bursts of ~maxing out the line rate.
Cilium is also running without host routing enabled but with vxlan and bpf masquarade.

1. High incoming packet loss on the physical card.
2. High outgoing packet loss on the lxc interfaces (host side of veth pair for containers).
3. bpftune ends up bumping netdev_budget and netdev_budget_usecs to frankly unreasonable numbers (nearly 300ms for some nodes), in an attempt to alleviate this.

The answer here has actually been to increase netdev_max_backlog. I'm not 100% sure but I think there are a few reasons this had a larger impact compared to the budget:

1. veth and all the other virtual adapters, as created by the cni, only have one rx and one tx queue. It's my understanding that this makes all container flows coalesce to one core's max_backlog, as the traffic exits the containers and goes down the chain of adapters.
2. Because of how deep the chain of adapters is (veth -> cilium_host -> cilium_net/cilium_vxlan -> physical card), the backlog and the budget are not necessarily evenly spread through all the devices, so it's easy for bursts in one to starve others, especially as they're packed on a single core (if my understanding is correct).

Increasing the backlog allowed us to pay with memory instead of softirq time and was far more effective than bpftune's slow spiral up in budget.

You folks would know better how to reason about this but in general it now feels important to me that bpftune also think about the depth of the backlog and potentially even the RPS/XPS configuration than just the budget limits.

[alan-maguire]

great info here, thanks! I added some rough heuristics to limit budget increases, but it sounds like they are insufficient here. Good point about separating the concept of budget from backlog. We do bump max_backlog too, would be interested to understand why that's not happening in your case. See https://github.com/oracle/bpftune/blob/main/docs/bpftune-net-buffer.rst for a description of what we are supposed to do (whether it's working right or not is another question!).

Anyway great food for thought in how we could better manage the relationship between budget and backlog. Currently the idea is that we limit budget increases by checking if the increases in budget correlate with longer wait-to-run times for processes (i.e. we are pushing out task execution). Ideally we could use the osnoise stuff that was added here but that's pretty recent in kernel terms. It would be good to see if I can add some stress tests that probe this functionality more with a view to tweaking things to avoid the runaway situation you've encountered.