Wgpu22 23% slower than wgpu 0.20 on my render-bench.

[John-Nagle]

Description
Wgpu22 seems to be 23% slower than wgpu 0.20 on my render-bench.

Repro steps
Build and run https://github.com/John-Nagle/render-bench/ branches "hp" vs "wgpu22safe". The second one uses WGPU 22, and all the more recent crates it requires. Build with

cargo build --release --features tracy
Start up Tracy Profiler 0.11.1 and click "Connect". It will then wait for the traced program to start.
./target/release/render-bench

Run for about a minute, with the graphics window on top and full screen, then close the graphics window.

When you run the program, it prints frame time stats. Every 10 seconds, it adds or deletes a large number of objects. Look at average time after adding objects.

Expected vs observed behavior
Frames average about 23% slower with WGPU 22.

There is almost no variation in frame time with WGPU 0.20, but there are slow frames a few times per second with WGPU 22. So there's a jank problem. All frames are slower, as well.

At least part of the problem is an intermittent lock stall in VkQueuePresentKHR. This is not the dominant user of time but seems to be responsible for the jank problem.

Note that this is a benchmark test, so that the developers can see this problem in a clean situation. My real program, the Sharpview metaverse viewer, has these problems, and they're much worse there. Occasional frames as slow as 200ms. So bad that WGPU 22 is unusable. Sharpview has a lot more going on in other threads than this benchmark, which only has two threads, one of which is idle except when it changes the scene every 10 seconds.

That's in wgpu/wgpu-hal/src/vulkan/mod.rs near line 1259. Nothing jumps out at me in "Blame"

Extra materials
Tracing results follow in updates.

Platform
Linux 22.04 LTS. NVidia 3070.

[John-Nagle]

Pictures of traces:
WGPU 0.20, one frame.

Note lack of red bars at top. All frames have consistent time. For this part of the run, render-bench is just refreshing the screen, with no changes.

WGPU 22 trace.

Note red bars at top. Those are slow frames. They're intermittent. This is a source of jank in a simple situation that should be jank-free.

[John-Nagle]

Zoom in on the jank trouble spot.

Most of this seems to be outside WGPU. Yet all that was changed was to convert Rend3 to use WGPU22, with the required version changes for EGUI and WINIT. This benchmark doesn't use EGUI, and WINIT isn't doing anything in the middle of a frame.

On a few frames I saw what seemed to be in intermittent lock stall in VkQueuePresentKHR. That didn't happen with WGPU 0.20. But it's not where much of the time is going.

Did something get bigger that's forcing cache misses?

[John-Nagle]

(All this was tested on desktop, not WGPU.)

[ErichDonGubler]

Regarding locking, there were some changes after 0.20.* in which we discovered and fixed some UB with Vulkan synchronization. They're in the CHANGELOG, if you'd like to find them (I'm on mobile, so linking takes more time than I'd like 😅).

[John-Nagle]

Where is that? Looking in https://github.com/gfx-rs/wgpu/blob/trunk/CHANGELOG.md and not finding any relevant changes later than 0.19, which is before this regression test.

[ErichDonGubler]

Ah, shoot, I made an off-by-one error; the PR I was thinking of is @cwfitzgerald's #5681, but that should be released as of 0.20.

🫤 I don't think I have any relevant information any more, sorry!

[John-Nagle]

OK, thanks.

For some slow frames, there's about 0.5ms of stall at the beginning of the frame, in vkQueuePresentHR. Usually, that's so fast it barely shows up in profiling. But sometimes it takes longer. No idea why. The program at this point is rendering the same image over and over. That's a source of jank.

That's not where most of the excess time is going, though. Some time goes into frustum culling and depth-sorting objects. Sorting does not seem to be slower, though, so although that may be a performance issue,it's not this problem.

There's a call to .get_mapped_range_mut(). Is it possible to get a stall there if something is happening GPU side? I need to add more profiling calls in Rend3 to pin this down.

[John-Nagle]

Much of the additional time is going into set_bind_group. Did that get slower?

[teoxoy]

It probably did due to #5874 but with #6419 (not yet part of any release) it should no longer be an issue.

[John-Nagle]

From #6419:

The problem here as far as I can tell is that we are creating 160k unique bind groups that all share the same texture in a loop.

which my render-bench is doing. So that's likely to be at least part of the problem. Thanks.

How soon can I test against something with #6419: and find out if this really is the problem? While render-bench does do that, the actual application has many more different textures. So there might be something else as well.

[teoxoy]

There will probably be a release this week.

[John-Nagle]

Oh, good. Please let me know when there's a branch I can test. Thanks.

[teoxoy]

Actually #6419 is already on trunk if you want to test it early.